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TRAINING DEVICE DESIGN: HUMAN FACTORS 
REQUIREMENTS IN THE TECHNICAL APPROACH 

ABSTRACT 

This report presents guidelines for achieving the human factors inputs 
to the Technical Approach in the training device "design process* A method 
is provided which facilitates the correlation of instructional requirements 
with engineering design solutions. Techniques and procedures are recom-, 
mended for organizing the information requirements which must be accounted 
for in the engineering design in order to maximize the instructional potential 
of a device* 

Three major sections are provided. The first of these presents techniques 
and procedures for deriving the information requirements relative to achiev- 
ing simulation fidelity in trainee station design. 

The second section presents procedures for deriving the information re- 
quirements involved in setting up, controlling, monitoring and evaluating 
performance at theinstructor station. Fourteen chapters describe the in- 
formation requirements pertinent to the structure and control of training 
during the off-line, pre-mission, enroute training and post-exercise 
operations. 

The last section discusses the human factors test and evaluation require- 
ments in the training device acceptance process. Procedures are outlined 
for verifying the suitability of a training device as an instructional system. 
The test, evaluation and demonstration requirements throughout device 
fabrication are organized to assist the human factors specialist in deter- 
mining that the device performs as advertised. 
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FOREWORD 



A previous report in this series "Human Factors Inputs to the Training 
Device Design Process", NAVTRADEVCEN 69-C-0293-1, discusses the Human 
Factors roie in the specification of requirements for training devices and 
presents substantive information for use in this process. This present 
report takes up where the previous left off. The interactions between 
Engineering and Human Factors specialists in trainee station and instructor 
station design and evaluation are described and exemplified. 




VINCENT J. SHARKEY Q 
Scientific Officer. 
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SECTION I 



INTRODUCTION 

1. INTRODUCTION 

1.1 BACKGROUND. The Naval Training Device Center Technical Approach 
determination (TA) provides the technical description of a training device in 
response to the Military Characteristics (MC) description of the device. 
Whereas the MC provides the initial documentation and describes the train- 
ing need and the training situation (NAVTRADEVCEN INST. 3910.4, 1969), 
the TA provides for the evolution of the technology (NAVTRADEVCEN INST. 
3910.5, 1970). Both accomplish their goals through the incorporation of 
concept, detailed and production cycles (i.e. , the concept TA responds to 
the concept MC, the detailed TA responds to the detailed MC, the production 
TA responds to the production MC). This process has been developed to 
provide a properly planned, described and costed presentation for a pro- 
posed training device. The training device development chronology is thus: 
MC, TA, procurement package. All phases require substantial human 
factors support and consultation. 

The Technical Approach is utilized to provide specific engineering 
concepts, development plans and cost estimates prior to the preparation 
of the procurement package for the training device. The documentation 
describes how the device will be developed, fabricated, acquired and sup- 
ported. Developing the Technical Approach (and preparing the report) is the 
responsibility of the engineering group. The objective of a TA is to develop 
sufficient engineering details and alternatives to enable a training agency to 
evaluate systematically the merits of funding a training device. * The TA 
phase is predicated on an existing data base; the functional training charac- 
teristics required to provide the desired training capability have already 
been derived. 

At present, there is no routinely used acceptable human factors pro- 
cedure for determining the technical approaches to training device design 
(i.e. , detailed information on how a device should be configured to achieve 
the desired training approach). Each device is handled in terms of its own 
objectives and frequently, the human factors inputs are inadequate or 
incomplete. 

1.2 PURPOSE. This report presents a guide for achieving the desired 
human factors inputs to the Technical Approach determination in the training 
device design process. It centers on the issues of human factors responsi- 
bility in device design, specifically, on the instructional requirements that 
must be fulfilled via engineering implementation. The human factors role 

is to specify the information requirements that must be accounted for in 
design in order to maximize the instructional capability of the training device. 



NAVTRADEVCEN 71-C-0013-1 



A method (techniques and procedures) is provided for systematically 
examining the relationships between alternative training system configura- 
tions and desired student performance. This method is set in a framework 
which facilitates the correlation of instructional requirements with engineer- 
ing options in the training device design process. In addition, guidelines 
are organized for identifying and examining the relevant design alternatives 
at the man-machine interfaces to assure the most effective instructional 
capability for the costs involved. In essence, this report represents a 
means of solution testing for subsequent engineering design to insure an 
instructional capability consistent with the training purpose and the training 
objectives articulated for a proposed device. The organizing methodology 
outlined in this document may be viewed as a contribution to the design 
process. 

1 . 3 PERSPECTIVE. The techniques and procedures described in this 
report are designed to support the engineering group by providing means 
for assuring that proposed engineers g solutions will optimize the instruc- 
tional capability of a contemplated training device. Guidelines are organized 
to assist the human factors specialist in interacting substantially with engi- 
neers in deciding on the utility of alternative technical approaches and on 
the final design selection. These guidelines provide a structure (design 
pathway) for the development of the information requirements for simulation 
fidelity in trainee station design and for achieving an instructional capability 
for shaping desired behaviors via instructor station design. Specifying the 
information requirements for achieving effective instruction is the primary 
theme erf the human factors support. This report provides assists which 
enable the human factors specialist to "arm himself" with the information 
requirements that should be accounted for in engineering design prior to 
any formal interactions with the project engineer in developing the technical 
approach documentation. In other words, the content of this report is devoted 
to the various human factor* analyses that should be completed as a prelude 
to formal interactions between engineering and human factor* in examining 
alternative engineering solutions for achieving the desired instructional 
capability within cost and lead-time expectations. This emphasis is most 
consistent throughout the report; the additional interaction requirements 
are merely outlined, leaving the actual design process to be accomplished 
within the procedures established by the engineering codes. This is in 
keeping with the support role provided by human factors in technical approach 
documentation. 

The procedures and formats described here pertain to the development 
of the technical approach for any contemplated training device. As such, 
they may at times suffer in desired specificity for a given design effort. 
However, the guidelines approach enables the known facts and the constraints 
to be systematically arranged for most design situations and this provides 
a basis for examining engineering alternatives for achieving instructional 



NAVTRADEVCEN 71-C-0013-1 



strategies consistent with design concepts for a range of training device 
classes. The feature that the human factors design analysis and tradeoff 
determinations are organized prior to any formal interaction with engineers 
makes possible the most economical and fruitful subsequent interaction for 
examining the relevant engineering solutions to defined training problems. 

♦ T1 2u ,t * rti " g P ° int f ° r thi * development is the MC Documen- 

to T\J * te ^ nique8 and Procedure, described in this report are anchored 
to the MC for the contemplated device and implicitly assume, a consistent 
referral to the MC for design direction. Thus, it is tacitly assumed that 
the appropriate background effort specifying the functional requirements for 
a contemplated training device has been completed as a prelude to the TA 
documentation This includes: the articulation of the purpose of the train- 
ing device; a description of the nature of, and the performance character- 
istics of the counterpart operational system; the task structure and training 
objectives; gross device definition; the functional characteristics of f e 
trainee station environment and the instructor station requirements; the 
anticipated student population; and the training schedule and the anticipated 
total instructional time allotted per student or team. 

A basic question concerns the manner in which human factors inputs 
are made to the Technical Approach documentation. In essence, human . 
factors support is provided the project engineer through the systematic 
analysis of -the information requirements for achieving the desired instruc- 
tional capability in the following: 

Trainee station— this effort considers the extent to which task 
elements must be simulated, their relative importance and 
criticality for training (clarification and the setting of the limits 
' of fidelity) to the detail required to determine coat and lead-time 
estimates in software and hardware. This includes a justification 
for the provision of the selected simulation elements vis-a-vis 
instructional value. Where applicable, an examination is also 
made of the means available (i.e., candidate engineering solutions) 
for best meeting the simulation requirements. 

Instructor station-this effort considers the display, control and 
communications requirements for the problem installation, moni- 
tor, control, evaluation and critique functions involved in training. 

How this is accomplished is treated in detail in the subsequent sections of 
this report. The outputs of this effort are incorporated by the engineering 
group into the technical approaches deemed feasible for achieving the desien 
requirements specified in the MC documentation. 
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1 # 4 ORGANIZATION OF THE REPORT* This report is written expressly 
for individuals charged with the responsibility for human factors inputs dur- 
ing the conception and development of the design of training devices* It is 
also envisaged to be of interest to simulation engineers involved in develop- 
ing the technical approach and the subsequent specification package for any 
contemplated training device. 

The material presented does not conform to the table of contents of 
NAVTRADEVCEN INST 3910-5, Preparation of Technical Approaches for 
Training Devices. Instead, it is organized as a logical approach for achiev- 
ing the human factors requirements specified in the instruction and corre- 
sponds most closely to the order in which the design effort should actually 
be accomplished* It also provides considerable detail in technique and 
procedure for achieving what we consider to be the requisite human factors 
contributions to TA documentation as specified by the gross categories out- 
lined in the Instruction. 

Three major sections are presented in addition to this Introduction! 
Section II presents procedures and techniques for deriving the information 
requirements relative to trainee station design; Section m organizes the 
control, display and communications requirements relevant to instructor 
station design; Section IV presents the information requirements for the 
evaluation of training system design. 
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section n 

TRAINEE STATION DESIGN 

2. INTRODUCTION 

An output of the MC defines the extent and levels of simulation needed 
to achieve the purpose of the training for the contemplated training device. 
This includes decisions on those portions of the operational system counter- 
part which" will be represented, the training system configuration, the 
system suites (i.e. , core equipments needed to operate analogous to the 
operational system), the device operating modes, as well as the design 
considerations associated directly with providing flexibility in developing 
the required instructional strategies. 

The effort now is to specify precisely the simulation requirements and 
appropriate fidelity levels for the contemplated training device. Decisions 
on what must be simulated and the design options available within the state- 
of-the-art together with costs constraints are anchored to the functional 
specifications of the MC* The design criterion is clear: provide the means 
for exercising and developing the human performances required in the op- 
erational environment. The human factors specialist is concerned with 
deriving the information requirements that must be satisfied in represent- 
ing the synthetic environment in which training is conducted. Trainee 
station design considers most prominently the issues involved in achieving 
simulation fidelity, that is, specifying the displays, controls and commu- 
nications involved in the device and the fidelity levels required at the man- 
machine interfaces, ranging from high fidelity in engineering simulation to 
deliberate departures from reality involving special features of design not 
found in the operational system, in order to enhance learning. Of concern 
is the extent to which certain events must be simulated and their .relative 
importance and criticality for training. 

The human factors input centers on the definition of the simulation 
elements which should be accounted for^in design. These represent the 
parameters and their values (range, envelope, number) needed to accom- 
plish the desired training, hence govern the range and complexity of task 
installation in the device. The simulation elements which are controllable 
(manipulated or modified manually or automatically via the instructor sta- 
tion) determine the perceptual equivalence of the training environment and 
the operational situation. The ability to provide the desired training in the 
identified tasks is directly dependent on the availability and adequacy of 
these simulation elements in a training device (i.e. , which ones are selec- 
ted and how usefully they are represented). Shortcomings in the simulation 
of these elements define the shortcomings in the training capability of a 
device. The simulation elements are associated with the representative- 
ness and complexity of the tasks to be trained in the device. 
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Thus, the design effort centers on: 

a clarification and a setting of the limits of fidelity (i.e. , 
a correlation of engineering design with the MC require- 
ments for the contemplated training device). 

equipments and software requirem* nts that are specific 
to the training device. This concerns aspects of the 
training situation not dictated by the operational system 
(i.e. , equipment and display items provided in addition 
to the vehicle compartment or cockpit configuration for 
enhancing the instructional capability). 

Minimal analytic emphasis is placed on design features specified in the 
MC for which engineering design practices are relatively fixed so far as 
fidelity is concerned (regardless of the adequacy of the current engineering 
state-of-the-art). These include such capabilities as platform motion, 
radar land mass, and control forces simulation. Human factors contribu- 
tions in these areas cannot be made within the time frame of the technical 
approach documentation. Significant improvements in representing these 
and similar design options require additional and often substantial research 
(see Section IV). Thus, where needed human factors data are lacking, the 
design practices within the current engineering state-of-the-art are tacitly 
accepted so far as human factors support is concerned, and no additional 
input is provided beyond the decisions made in the MC/ 

2.1 RATIONALE FOR HUMAN FACTORS INVOLVEMENT IN TRAINEE 
STATION DESIGN. The human factors effort centers on the information 
requirements that mast be accounted for in any training device design 
approach. The engineer requires data which specify how much of and how 
well the tasks should be represented in order to determine the relevant 
alternative technical approaches, one of which will ultimately be selected 
for mechanization. The human factors contribution includes a depiction of 
how many parameters are needed and in what form in order to represent 
the task, i.e. , what simulation elements must be provided to achieve the 
purpose of training and the training objectives for the device. Now, the 
fidelity issues of what should be simulated in light of the training goals have 
already been examined in the MC (i.e. , what do you want to have happen?). 
For the Technical Approach, statements are needed setting the adequate 
levels of fidelity and clarifying the simulation requirements (i.e. , in what 
ways can the desired end results be obtained?). The relative importance 
or criticality of the simulation elements for training must be described. 
Critical simulation elements are those which exert an appreciable effect 
(based on the best available evidence) on rate of learning or on level of 
proficiency. This information on fidelity requirements is provided in the 
detail needed (i.e., the extent of , and criticality of the simulation elements 

ERIC -6- 
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to be represented with a justification of why an element is important for 
training) to permit costing and lead-time estimates to be made. Armed with 
these data, the human factors specialist can support the engineer in justi- 
fying'the inclusion of the relevant simulation elements for the costs incurred, 
and in decisions on optimum ways for mechanizing those requirements in- 
cluding, as applicable, an examination of candidate engineering solutions. 
What to mechanize must be finally selected in terms of costs and lead-time 
vs. training effectiveness (defined by the ability to meet or extend the 
achievement of the training objectives outlined in the MC) . 

2.2 BASIC APPROACH. Various techniques and procedures are provided 
as guidelines for analytically determining the information requirements and 
the instructional capability that must be represented in design. For human 
factors, fidelity has meaning in terms of the training process and the realism 
necessary to promote transfer of training. Defining the design. character- 
istics for maximizing transfer of training from the synthetic environment 
to the operational requirement revolves essentially about two interrelated 
questions: what or how much should be simulated, and how well should this ■ 
be represented in terms of degree of physical correspondence to the opera- 
tional environment? These two questions plus the concern for incorporating 
training, technology into design (integrating training process with engineering 
fidelity) circumscribe the approaches to fidelity. The fidelity decisions 
must necessarily consider the monetary costs and the lead-time problems 
entailed and what the omission of critical simulation elements will cost in 
training effectiveness. Some assistance in this determination is provided 
in this document. 

In essence, guidelines are provided the human factors specialist by 
means of detailed techniques and procedures which serve in organizing the 
simulation elements crucial to engineering design. These information re- 
quirements formats, once completed, can be effectively employed by the 
human factors specialist in interacting with engineers to examine candidate 
engineering solutions and to compare relevant technical approaches in 
terms of training effectiveness, costs, and lead-time constraints, These 
techniques and procedures are described in detail next. 

2.3 TECHNIQUES FOR ACHIEVING HUMAN FACTORS INPUTS TO 
TRAINEE STATION DESIGN. The basic theme of the human factors effort 
is to determine the information requirements which engineering must account 
for in proposing alternative technical approaches to trainee station design. 
This concerns the definition of the simulation elements to represent and 
the criticality of these elements to training, the levels of fidelity to imple- 
ment and specifying techniques for instructional control in order to achieve 
the training capability defined for the training device. Four variations on 
this basic theme are described.below. • These provide- techniques and pro- 
cedures that are pertinent to a range of training device design situations. 
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The need for this variation recognizes that information requirements for 
design will vary as a function of classes of training device both in terms'of 
the operational requirements (e.g., flight, surface, subsurface) and the 
trainee personnel (e.g. , multiple and interdependent students such as in 
tactical team training; individual students such as in an OFT; and multiple 
but independent students such as in group training for a given specialty).. 
Thus, the four distinctive techniques provide guidelines for deriving the 
human factors simulation requirements applicable to a range of training 
device design situations. The reader needs but to discern which of the 
cases is applicable to his specific design strategy and apply the guideline 
procedures, as applicable. 

The four techniques are the following: 

Case 1: Technique for deriving information requirements and 
setting the limits of simulation fidelity. 

Case 2: Technique for correlating candidate, engineering 
solutions with simulation requirements 

*} 

Case 3: Technique for defining display and control require- 
ments in generalized trainee station design. 

Case 4: Procedures for employing instructional technology 
to enhance training capability. 

2.3.1 CASE 1: Technique for deriving information requirements and 

setting the limits of simulation fidelity. 

A basic human- factors effort is to identify the simulation elements 
to be represented for training, for these define the device hardware and the 
software programing requirements. At stake are the following: How much 
to simulate and what levels of fidelity to provide (with the implicit cost- 
effective question of what may be omitted from complete fidelity of repre- 
sentation without significant decrease in training value). The simulations 
engineer needs to know what and how many parameters must be simulated 
and the fidelity requirements in order to adequately represent the task for 
training . 

The identification of simulation elements (i.e. , those factors to be 
simulated that determine the representativeness and the complexity, of the 
tasks to be installed for training), and the parameter values (range, envelope, 
number), are crucial inputs to the definition of fidelity. These must be 
clearly specified in order to achieve the required device configuration to 
enable training to the level expected. In short, the simulation elements 
define what is needed in the synthetic environment to achieve the expected 
transfer of training. The output of this effort is an organization of simulation 
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elements that are relevant to achieving the training purpose defined for 
the contemplated training device. 

The procedure for accomplishing the derivation of the design in- 
formation is outlined in the following series of steps: 

1. Identify the design subsystem or area 

2. Identify the training problem — define the task requirements 
associated with the design subsystem or area 

3. Identify the critical simulation element? (information require- 
ments, mission derived); correlate these with acceptable 
levels of fidelity. 

4. Describe the training effectiveness implications of each of 

the decisions in Step 3. (The procedure tlms far is* accomplished 
by the human factors specialist; 'from here, the human factors 
specialist interacts substantially with the project engineer). 

5. Interact with the project engineer to: justify the simulation 
elements selected and the fidelity level for the element; corre- 
late the information requirements with alternative technical 
approaches and engineering implementation; and compare the 
technical. approach choices with cost and lead-time considerations. 

6. Support the engineer in documenting the proposed technical 
approach for the given design subsystems or areas. 

These procedures are described in detail next, together with an 
example selected to provide the necessary analytic guidelines for the reader. 

Step 1: Identify the subsyst em on operational counterpart to be 
simulated 

The pertinent subsystems involved in any contemplated training 
device are identified in order to systematically examine the simulation re- 
quirements. Within this frame, the major subsystem groupings are organ- 
ized. Individual analyses are then conducted for each subsystem (as 
appropriate): sonar, radar, electronic warfare, visual real-world, vehicle 
control, and weapons in order to derive the simulation requirements speci- 
fic to each. Within each of the relevant subsystems, the simulation elements 
are analyzed and organized independently in terms of: own-vehicle and 
support units; targets; and media characteristics. Figure 1 depicts these 
relationships in matrix fc^m. 
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Step 2: Identify the task requirements associated with the design 
subsystem or operational counterpart 

The task structure provides the clues for determining what must 
be simulated for training. The range of task requirements influences the 
decisions on what and how many parameters should be installed in the 
training environment* Table 1 presents a major grouping of task structure 
(involving mission/ vehicle combinations) appropriate to the design areas 
identified in Step 1. For each of these task groupings, detailed task analy- 
sis data are required in order to specify the simulation elements required. 
(It is assumed that this detailed information is available from the training 
requirements analysis conducted during the^Military Characteristics docu- 
mentation phase). 

Step 3: Identify the simulation elements to be represented and 
correlate these with acceptable levels of fidelity' 

The MC documentation serves as the basis for this essential step 
in the technique. Training purpose , task Structure and the training objec- 
tives have already been articulated and decisions have been made on the 
training system configuration, the system suites that will be represented 
(e.g. , oQS-23 Sonar, Mk 114 Fire Control System), and on the basic fidel- 
ity levels desired for classes of training tasks. Since these features of the 
device are set, specifying the parameters and their values (in hardware 
terms) which comprise the synthetic environment and their fidelity levels 
is the next order of business. The identification of the simulation elements 
gpveriB the range and complexity of task installation in the device and thus 
influences training effectiveness (i.e.', the achievement of training for the 
defined task structure) . The ability to provide the desired training in the 
identified tasks is directly dependent on the availability and adequacy of 
these simulation elements in a device (i.e. , which ones are selected and 
how usefully they are represented) . 

The array of the simulation elements that must be evaluated for 
representation in the design of the trainee station are organized in terms 
of own- vehicle and support units, target, and media characteristics. With- 
in each of these three groupings, decisions must be made about the dimen- 
sions or levels of fidelity required. 

Four levels of fidelity are employed in specifying the character- 
istics required to insure effective task installation. 

Level 1 — is defined by high fidelity representation (where the 
engineering state-of-the-art is adequate).. This involves two 
aspects: 
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TABLE 1. MAJOR CLASSES OF TASKS APPROPRIATE TO THE 
ANALYSIS OF SIMULATION REQUIREMENTS 
j " * : 

|l. SURVEILLANCE 

a. Tatrget acquisition (search, detect, classify, track) 

b. Signal analysis 
2. VEHICLE CONTROL 

a. Airborne 

1) Air-to-air environment 

• refueling 

• combat 
formation flight 

• ASW station-keeping 

2) Air-to-ground environment 

• visual navigation 

• reconnaissance 

terminal area (take off/ landing) 

3) Air flight 

• maneuvers 

• vehicle positioning 

. b. Surface 

1} harbor navigation 

2) mooring/ docking 

3) vehicle positioning 

c . Subsurface 

1) approach/ attack positioning 

2) navigation/ reconnaissance 

3) casualty/ damage control 
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TABLE 1. MAJOR CLASSES OF TASKS APPROPRIATE TO THE 
ANALYSIS OF SIMULATION REQUIREMENTS (Continued! 

3. WEAPONS DELIVERY 

a. Air -to -surface 

b. Surface-to-air 

c. Surface -to- subsurface 

d. subsurface -to- surface 
e # Air-to- Air 

f • Surface-to-surface 

g. Subsurface-to-subsurface ■ 

4. TACTICAL EMPLOYMENT 

a, Tactical decision-making 

b. group/ force deployment in tactical engagement 
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• precise representation of the operational system counterpart 
(envelop, steps, values, range) 

• deliberate backing off from high fidelity (to achieve cost 
economies without compromising training effectiveness) 

Level 2 --is defined by the fidelity achievable when reduction in 
tolerances is required (where the engineering state-of-the-art 
is less than adequate). 

Level 3- -is defined by the simulation of generalized/ universal 
functions (which may involve low fidelity representation as well 
as part-task simulation). 

Levef4--is defined by deliberate departures from realism in 
order to enhance training effectiveness (qualitative deviations 
from the operational system being simulated). 

The derived critical simulation elements are correlated with the required 
fidelity levels, using the format outlined below. 



Critical 
Simulation 


Fidelity of .Simulation 
" Levels 


Elements 


' 1 


2 


3 


4 


(Task/ vehicle 
combinations) 










• Ovn-vehicle and 
support units 










. Target • 










• Media 











The classes of simulation elements are grouped within each major sub- 
system applicable to the device under consideration. 

An example of the analytic derivation involved in Step 3 is described 
next. The example selected concerns the information requirements involved 
in Sonar Subsystem design (see Step 1). The task structure involves sur- 
veillance requirements in an ASW tactical team training context (mission- 
vehicle combination: Destroyer (DLG, DDG) ot Cruiser in ASW operations) 
(see Step 2). Table 2 presents the derived critical simulation elements 
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Level of Fidelity Required 


Deliberate 
departures 
from realism 
(qualitative 
deviations 
from the oper- 
ational system 
being simu- 
lated) 


* 


Simulate 

generalized 

functions 




[Fidelity 
[achievable 
j where re- 
duction in 
tolerances is 
required (en- 
gineering 
state-of-the- 
art is less 
than adequate) 




High fidelity 
(engineering state-of- 
the-art is adequate) 

... * i 


appropriate video/audio 
characteristics 

appropriate video /audio » 

tion of speed increase 
correlated with optimum 
sonar speed; appropriate 
audio degradation as a 


Critical simulation elements 
(information requirements) 

Example: Sonar Surveil- 
lance Task 
(surface ASW 
vehicle) 


A. OWN VEHICLE AND 
SUPPORT UNITS 

1* Own-vehicle 
returns 

• own-ship noise 
(flow noise, 
turbine noise) 

• wakes 

• knuckles 

• lorpeao nyuropnone 
effects 

• counte rmeasur e s 
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correlated with the required fidelity levels for the Sonar subsystem involv- 
ing surveillance requirements in ASW operations. This provides guidelines 
for a procedure that must be accomplished for each of the. major subsystems 
pertinent to the contemplated training device. 

Step 4: Document the training effectiveness implications of the 
simulation elements selected for representation 

Justification is now provided concerning the instructional advantage 
of the simulation elements selected and the acceptable, fidelity levels. This 
discussion should also indicate any disadvantages accruing from eliminating 
certain information requirements and what effects reducing or enhancing 
the fidelity level will have on the instructional capability. Those simulation 
elements crucial to the training problem should be identified (i.e. , those 
key information items, if rejected or modified significantly, will impair 
the instructional purpose of the device, and thus should not be compromised). 
In addition, the human factors specialist should be prepared, in discussions 
with engineers, to modify or eliminate selected simulation elements, 1) which 
may be logically pertinent to the trainee station design, but be of indeter- 
minate training value, 2) which will enhance training value but when weighed 
against costs to obtain, are excessively expensive to mechanize, or 3) which 
are not desirable for other than technical reasons. 

The justification should be anchored to antecedent MC decisions 
in terms of overall device configuration, the basic ff core ,f equipments and 
suites, and the fidelity levels envisaged for the trainer. The MC decisions 
affecting our sonar subsystem example include the following: 

• Sonar is a component of an ASW team trainer, 

• Training emphasis is on precedural and tactical decision- 
making sequences involving coordinated team effort in 
operating and employing a surface vehicle, 

. Specific system suites are required since a direct transfer 
of training is desired from the device to the operational 
vehicle (teams will be drawn for training from operational 
units when in port), 

• Compartment spaces, equipments and operating modes are 
replicas of equipments aboard ship. The physical appear- 
ances of key equipments with all operating controls and 
indicators are the same. For the present example, the 

relevant equipments are: the AN/SQS-23 sonar and the ^ 
Mk 53 Attack Director Console, C J) 
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. The information requirements are based on a high fidelity sonar 
environment with signal characteristics realistic of the range 
of real-world events, 

. Device configuration must provide a training environment con- 
ducive to accomplishing the following: procedures in search 

. and detection; maintenance of contact and tracking; procedures 
for readying weapons system for attack; utilization of commu- 
nication circuits; and procedures in the prosecution of targets. 

A format for justifying the derived simulation elements for our 
sonar subsystem" example is shown below. Only key concepts and require- 
ments are identified. In actual situations, the narrative should be expanded 
and include any supporting evidence. The goal is. to maximize the flexibility 
for training in achieving the relevant training objectives. 

!• Own-vehicle requirements 

The elements identified will provide the necessary realism in 
sonar displays for achieving the training objectives specified in the MC. 

Provision of own-ship wakes and knuckles gives CIC the oppor- 
tunity to account for these phenomena and to coordinate with sonar to pre- 
vent acquisition and tracking of these false targets. 

Sonar returns sensitive to vehicle size or aspect are cues to 
classification. 

Target classification capability is crucial to realizing the full 
potential of the trainer. Classification (false target capability) introduces 
the element of uncertainty in threat evaluation, requiring decisions for deaU 

attendant time losses in 
determining if the contact is a possible submarine. Lack of classification 
uncertainty modifies team procedures away from the realism of real-world 
operations. The presence of classification cues forces a greater decision 
making requirement on the team and a greater tactical involvement. Sonar 
subteam skills are also enhanced due to greater whole-task replication in 
the device resulting in increased involvement and responsibilities. 

2. Target 

Both conventional and nuclear submarine vehicle dynamics are 
required (e.g. , equivalent of Russian "Fox Trot" and "November" vehicles). 

' ^ . A . ccelerati ° n s and decelerations of nuclear submarines must be 
realistic in view of the tactical geometry requirements. 
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Maximum detection ranges corresponding to the full range 
achieved operationally (realistic as a function of sea state, layer depth 
and submarine depth and aspect) are mandatory in order to provide neces- 
sary training in use of stand-off weapons systems. 

False targets and submarine wakes contribute to the classifica- 
tion requirement. With a display of wake, the sonar operator may errone- 
ously "track off" on this trail. It also provides an evasive capability for 
the submarine in training exercises which emphasize tactical deception and 
greater own-ship threat evaluation*' 

Sonar video or audio failures enhance the control of exercise 
difficulty* A complete sonar failure, for example, would force own-ship 
. to execute a DEEP CREEP attack (own- ship vectored in). 

3. Media 

Sea state range correlated with wind speed will cause degrada- 
tion on sonar (reverberations). Also, a lack of correlation will produce 
unrealistic events (e.g. , sea state 1, wind speed 22 knots). 

Providing a controllable bottom depth from 100 feet to 2000 feet 
and showing bottom characteristics as well as depicting representative 
harbor areas permits the achievement of training objectives concerned 
with shallow water operations. 

Absence of non-target returns (false targets) also obviates the 
necessity for the use of bottom charts and CIC keeping track of underwater 
obstructions* 

4. Summary 

For any device situation a clear and precise justification for 
the simulation requirements is documented, vis-a-vis training flexibility 
and effectiveness* Again, in the above sonar example, key design features 
include: 

• video simulation — no cleaner nor less flexible than in 
the operational environment 

• capability for deliberately degrading equipment per- 
formance 

# high fidelity signal characteristics 

# provision of the parameters associated with uncertainty 
in target identification (classification function). 
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This procedure (Case I) is replicated for all of the pertinent 
subsystems in the contemplated training device. When completed and in- 
tegrated, the number of-, and values for the simulation elements have been 
identified sufficient for installing the environment for training. 

NOTE: The steps described thus far are 
accomplished by the human factors specialist 
utilizing the MC documentation and other data 
available on the contemplated device. When 
the information requirements analysis is 
completed, the human factors specialist works 
in close interaction with the cognizant engineer(s) 
to complete the procedures involved in deter- 
mining the simulation requirements in trainee ■ 
station design. Thus, beginning with Step 5, 
the output reflects the combined efforts of 
human factors and engineering. 

Step 5*. Correlate the information requirements with alternative 
technical approaches for engineering implementation 

The human factors and engineering specialists interact to achieve 
the following. 

• The information requirements are reviewed and accepted, 
with revisions as appropriate. Justification for the final 
selection of the simulation elements and the appropriate 
levels of fidelity is documented. 

• The means available for achieving the simulation requirements 
are identified. This may result in a straightforward engineer- 
ing approach to the simulation requirements or in several 
feasible technical approaches involving candidate engineering 
solutions . 

• In the event that engineering choices are available to meet the 
simulation requirements, the capability of the alternative tech- 
nical approaches to meet the simulation requirements are 
analyzed. A matrix format is recommended which enables a 
comparison of each candidate solution with the simulation re- 
quirements. One way of accomplishing this is illustrated in 
Case 2 of this section. Each candidate technical approach 
must meet, as minimums, the critical simulation elements 
(i.e., those that are crucial to solving the training problem). 



-25- 



NAVTRADEVCEN 71-CU0013-1 



. For the engineering solutions capable of meeting the defined 
simulation requirements, costing and lead-time approximations 
are made. 

The design criterion is clear; provide the minimum fidelity of simulation 
consistent with the training requirements information. The appropriate 
design options are based on the MC requirements and on the candidate 
engineering solutions available to achieve the simulation requirements. 

Step 6: Provide support in preparing the technical approach 
- documentation r~ 

The human factors specialist provides support to the engineering 
code in preparing the pertinent sections of the Technical Approach docu- 
mentation as specified in NAVTRADEVCEN INST 3910.5. 

Summary 

Deriving the information requirements (simulation elements to be repre- 
sented) and the setting the limits of simulation fidelity is one of the major 
human factors inputs to the technical approach documentation for trainee 
station design. The procedure for achieving this is concerned primarily 
with the human factors derivation of the simulation elements that must be 
represented in a device to achieve the purpose of the training as articulated 
in the MC documentation. The output of this effort is applied by engineering 
and human factors specialists as a basis for developing proposed technical 
approaches to device design. Figure 2 summarizes the series of steps 
accomplished in the procedure. 

2.3.2 CASE 2: Technique for Correlating Candidate Engineering 
Solutions with Simulation Requirem ents 

The visual simulation design requirement places a somewhat unique 
demand on the human factors input because the development of technical - 
approaches is tailored directly to the characteristics of basic visual simu- 
lation systems. 

A technique 1 is provided here for developing the simulation re- 
quirements information necessary for specifying the visual simulation 
equipment to be used for training. It is an analytic approach designed to: 
1) collate the scattered bits and pieces of information about any specific 



^his technique was propounded by Dr. KnoxE. Miller of the U.S. Naval 
.Training Device Center. A minor expansion of the original formulation 
is provided here. 
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visual simulation problem as seen from the human factors and engineering 
standpoints , 2) organize this information in a systematic (albeit arbitrary) 
manner, and 3) display the resulting data in a format which will permit 
direct comparison of the relative advantages and disadvantages of competing 
hardware systems. To apply the procedures, the human factors specialist 
must have a detailed knowledge of the constraints imposed on the hardware 
selection process by the following factors: a) the characteristics of the 
operational environment to be simulated; b) the perceptual requirements 
of the specific mission or mission segment involved; and c) the functional 
characteristics of the simulation hardware under consideration. The 
technique should be regarded as a convenient way for organizing and dis- 
playing information already available for design. In essence, it consists 
of a logically based fractionation of visual simulation requirements (derived 
from (a) and (b) above) on the one hand, and simulation system character- 
istics on the other-, into units of a size which will permit the known facts 
about each to be arrayed in a series of simple decision matrices. Thus, 
for each combination of a vehicle/ miss ion segment (e.g. , fixed wing, low 
altitude navigation), and a candidate simulation system (e.g. , CCTV model) 
a number of matrix "scores" can be derived, each of which represents, for 
a given characteristic, the degree to which the simulation system in question 
meets the requirements imposed by the specific environment/ mission to be 
simulated. When the individual matrix scores are graphically displayed as 
a function of the arrayed simulation requirements, a profile is obtained in 
which the relative strengths and weaknesses of the simulation system under 
consideration are made readily apparent. Comparison 'of the profiles de- 
rived for each of several candidate simulation systems as a function of the 
same set of simulation requirements thus provides a logical and objective 
basis for the selection of hardware. 

The procedure for comparing alternative design solutions to the 
visual training problem is outlined in the following series of steps. 

1. Identify the visual simulation problem. 

2. Identify the candidate engineering approaches for solving 
the visual simulation problem. 

3. Describe the visual system characteristics which bear on 
the hardware selection process. 

/ 

4. Construct a series of matrices correlating visual simulation 
requirements with system characteristics. (Basically, the 
procedure thus far is accomplished by the human factors 
specialist; however, he may interact with engineering in 
achieving Step 2 requirements. Step 5 is a correlative 
effort between human factors and engineering specialists.) 
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5. Develop a system characteristic profile for each candidate 
engineering approach, depicting the matrix scores for each 
visual system characteristic. 

These procedures are described in detail below, together with examples 
in order to provide the necessary analytic guidelines for the reader. 

Step 1: Identify the visual simulation probl em 

This involves the selection of the visual simulation problem from 
a grouping of visual problems pertinent to the NTDC training device in- 
ventory. Although the number and variety of potential visual simulation 
problems is large , those associated with vehicular control are most 
prominent. These can be logically classified and organized in terms of 

a) the operational environment (e.g. , air-to-ground, waterborne, etc.), 

b) the type of vehicle (fitted- wing, surface ship, etc.), and c) the mission 
or mission segment (e.g., low altitude visual navigation, harbor navigation 
etc*) which they represent. The remainder consist of artificial sensor 
display (radar, sonar) problems and a few special cases which are distinct 
.from, but so closely related to, visual simulation that they are often con- 
sidered under the same heading. The classification scheme adopted for the 
purpose of this analysis is presented in Table 3. This particular classifi- 
cation is useful because it permits the total mission for a given weapon 
system to be broken down into meaningful segments, each of which is well 
enough defined to be dealt with on an analytic basis. For example, a 
carrier-based attack aircraft on a strike mission might conceivably involve 
several of the individual mission segments shown in Table 3, each of which 
has implications for the simulation hardware to be employed. Simulation 
of the complete strike mission (including carrier takeoff and landing, low 
altitude visual navigation, target acquisition, weapons delivery, etc*) 
therefore, requires cascading of the analytically determined hardware con- 
straints associated with each of the mission segments involved* Such pro- 
gressive restriction of alternative solutions quickly reveals that there is 

at present no single technique capable of providing a satisfactory solution 
to the requirement for a total strike mission simulator. But more im- 
portantly, this technique can be used to reveal the extent to which various 
mission segments (part-tasks) can be combined in practical and reasonably 
cost-effective training equipment. 

Step 2: Identify the candidate engineering approaches for solving 
the visual simulation problem 

The number of engineering approaches available to solve the 
problems defined by the selected mission segments is limited* These 
approaches are listed in Table 4* This listing is not inviolate; the intent 
is to include all of the various technologies which have or can be used to 
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TABLE 3. MILITARY VISUAL SIMULATION PROBLEMS 



I. Vehicle Control Problems 
A. Airborne Vehicles 

1 . Air-to-Air Environment 

a. Fixed Wing 

( 1) Combat Maneuvering 

(2) Air-to-Air Refueling 

b. Rotary Wing 

( 1) Tactical Formation Flying 

(2) ASW Station Keeping 

2. Air -to -Ground Environment 

a. Fixed Wing 

( 1) Visual Navigation ' ' 

(2) Visual Reconnaissance 

(3) Target Acquisition 

(4) Weapons Delivery 

(a) Bombs 

(b) Visually guided missiles 

(c) Gunnery 

1. Fixed 

2. Flexible 

(5) Takeoff/ landing (fixed base) 

(a) Night 

(b) Day 

(6) Takeoff/ landing (carrier-based) 

(a) Night 

(b) Day 

b. Rotary Wing 
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TABLE 3. MILITARY VISUAL SIMULATION PROBLEMS (Continued) 

(1) Visual Navigation 

(2) Visual Reconnaissance 

(3) Target Acquisition 

(4) Weapons Delivery 

(a) VisuaUy guided missiles 

(b) Gunnery (flexible) 

c. Remotely Controlled (TV guided) Missiles 
B. Surface Vehicles 

1 ♦ Land Based Environment 
a. Off Road Vehicles 

(1) Tanks/tracked Vehicles 

(2) Air Cushion Vehicles 

b* Conventional Vehicles 
2. Waterborne Environment 

a. Surface(d) Ship Control Problems 

(1) Harbor Navigation 

(2) Slup/ submarine Mooring 

( 3) Ship/ submarine Docking 

(4) Ship-to-Ship Underway Replenishment 

b. Small Boats and Special Purpose Waterborne Vehicles 

( 1) Assault Boats 

(2) Hydrofoil 

C* Subsurface Vehicles 

1. Subsurface -to-Surface Environment (Periscope Simulation) 

a* Approach/attack 
b* Reconnaissance 
Cs Navigation 

2. Subsurface Environment (in- water simulation) 
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TABLE 3. MILITARY VISUAL SIMULATION PROBLEMS (Contir :ed) 



II. Artificial Sensor Display.Problems 

A. Radar/radar landmass 

B. Sonar 

C. .IR 

D. TV- Data Link 

E. ECM 

F. Compound types 

III. Special Information Display Problems 

A. Status and Large Area Tactical Information Displays 
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TABLE 4. BASIC VISUAL SIMULATION SYSTEMS 



1. 


Optically Viewed Models 


2. 


Closed-Circuit TV Viewed Models 


3. 


Point Light Source - Transparency 


4. 


Open -Loop Movie 


5. 


Servoed Projections 


6. 


Semi -Closed Loop (VAMP) Movie 


7. 


Computer-Generated Display 


8. 


Flying Spot Scanner - Factored Transparency 


9. 


Laser/Holographic Displays 


10. 


Hybrid Systems 



-33- 



NAVTRADEVCEN 71-C-0013-1 



generate, store , transfer and display imagery for the purposes of visual 
simulation. The successful application of this technique requires an ade- 
quate knowledge of the relative strengths and weaknesses of the simulation 
hardware available* At the very least, the human factors specialist must 
understand the major advantages and disadvantages of each of the competing 
systems under consideration (as a prelude to the. subsequent interaction 
with the engineering code)* 

It is not our purpose to present an exhaustive analysis and detailed 
comparison of the specific characteristics of each of these systems. Rather 
it is assumed that the reader either possesses or has available to him the 
background information required to relate these system characteristics to 
the perceptual requirements of the training situation. However, it is de- 
sirable from the standpoint of orientation to review briefly the principal 
features of each basic system. 

1. Optically viewed models 

a. Typical application - periscope view simulation. 

« 

b. Primary advantages - high resolution, closed-loop 
operation, realistic cues to depth, true colors available. 

c. Primary disadvantages - low brightness, narrow angle 
of view, difficult scaling problems. 

2. Closed Circuit Television - viewed models 

a. Typical applications - air-to-ground target acquisition, 
carrier landing. 

b. Primary advantages - moderately high resolution at 
moderate brightness levels, closed-loop operation, 
realistic depth cues, electronic ; manipulation of imagery 
possible (raster shrinking, video insertion, selective 
blanking, etc.). 

c. Primary disadvantages - narrow angle of view, diffi- 
culty *n achieving flexible scaling. 

3. Point Light Source and Transparency Systems 

a. Typical application - approach and landing simulators. 

b. Primary advantages - extremely wide angle' field of view 
available, closed-loop operation, full color easy to provide. 
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c. Primary disadvantages - low resolution and low brightness, 
poor depth cues, difficult scaling problems. 

4. Open - Loop Movie 

a. Typical application - low altitude navigation and target 
acquisition simulators. 

b. Principal advantages - high-resolution, high brightness, 
full color, extremely wide. angle of view, good depth cues 
available, scaling inherent in film. 

c. Principal disadvantages - open-loop operation, difficulty 
in acquiring adequately controlled imagery. 

5. Servoed Projections 

a. Typical application - Redeye Moving Target simulator. 

b. Primary advantages - high resolution, high brightness, 
extremely wide angle field. 

c. Primary disadvantages - reversed contrast of targets 
(target appears as bright image against dark background), 
open -loop operation. 

6. Semi-Closed Loop (VAMP) Movie 

a. Typical application - takeoff and landing simulators. 

b. Principal advantages - high resolution, high brightness, 
full color, good depth cues, scaling inherent in film, semi- 
closed loop capability permits limited maneuvering within 

a narrow envelope. 

c. Principal disadvantages - deliberately introduced optical 
distortion, difficulty of acquiring proper imagery. 

7. Computer Generated Displays 

a. Typical application - takeoff and landing, air-to-air combat 
maneuvering, ship docking and station keeping, etc. 

b. Principal advantages - closed-loop operation, scaling inher- 
ent in programing, independent motion available within visual 
field, resolution and color elements can be differentially 
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apportioned according to areas of interest, entire simulation 
problem can be changed or the elements within a problem 
updated by programing . 

c. Principal disadvantages - relatively high system and soft- 
ware costs i imagery consists of line drawings or discon- 
tinuous cartoon-like figures* 

8. Flying Spot Scanner - Transparency Systems 

a* Typical application - radar landmass simulators* 

b. Principal advantages - closed-loop operation, method of 
. reading out information (density along narrow beams se- 
quentially scanned by a small spot) produces very good 
analogue of a scanning radar beam* 

c. Principal disadvantages - resolution poor at low simulated 
altitudes, high cost of producing properly coded transpar- 
encies at scale factors required* 

9. Laser/Holographic Displays 

a* Typical applications - high brightness, large screen tactical 
situation displays, carrier landing simulators* 

b* Principal advantages - very high brightness , multiple colors, 
and variable persistence of photochromically recorded laser 
images makes them ideal for large screen status displays* 
Three dimensional quality of holographic imagery is valuable 
in simulating carrier landing, ship docking, etc* 

c* Principal disadvantages - narrow angle available with 
holograms, monochrome images , experimental nature of 
current technology* 

10. Hybrid Systems (employing a combination of two or more of the 
basic approaches outlined above)* 

• a* Typical application - visual mission, for example, simulator 
utilized in the A-7D weapon system trainer* This proposed 
system provides visual simulation for takeoff, low altitude 
navigation, target acquisition, weapon delivery and landing, 
by incorporating the basic characteristics of a transparency 
based terrain model, and a semi-closed loop (VAMP-type) 
projection system. 
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b. Principal advantages - high brightness, high resolution, 
full color, semi-closed loop capability. 

c Principal disadvantages - narrow angle of view, cumbersome 
solution to scaling problems (necessity for repeated alterna- 
tion of imagery stored in film cassettes), at least seven 
separate sources of distortion whose effects cumulate in 
unpredictable ways. 

The simplified summaries presented above can be questioned 
in specific instances. For example, it could be argued that TV-based sys- 
tems are not necessarily limited to narrow fields of view. However, in 
order to achieve a wide field, one must sacrifice both resolution and bright- 
ness or accept the problems inherent in achieving and maintaining image 
registration in multiple adjacent displays. Similarly, it could be pointed 
out that the narrow angle of view which is characteristic of optically viewed 
systems is not a serious disadvantage in periscope view simulations. In 
this case, the angle of view is a disadvantage of the general system type, 
though not in the application cited. ,p 1 

In listing the relative advantages of the various systems, two 
different types of features are cited. Features such as resolution, bright- 
ness, distortion and displayed angle of view are usually considered basic 
parameters of any visual simulation system. Other features such as the 
ability to provide independent motion of targets within the visual scene, the 
relative difficulty in making changes in the stored, imagery and whether or 
not the image storage medium is capable of providing adequate cues to depth 
are not ordinarily thought of as system parameters. Although the distinc- 
tion between them is somewhat arbitrary, it is a significant one since an 
important aspect of our technique is that it deals explicitly with those factors 
which are crucial » the selection of simulation hardware but which are not 
recognized as quantitative system parameters. These are the somewhat 
nebulous difficult to quantify, but highly important considerations which 
are usually a product of one's experience. As such, they often play a deci- 
sive but implicit role in the dec is ion -making process. The basic problem 
with such experientially-dependent decisions lies in the individual variations 
m the implicit definitions of these factors and in the relative weights 
assigned to each. By explicitly defining as many of the system character- 
istics which bear on the hardware selection process as possible, this anal- 
ysis enables the engineering team to arrive at their decisions on a more 
logical basis . 

Step 3: De scribe the visual system characteristics wh ich bear 
on the hardware selection process " — " 

. The vi8u u al 8 y 8 tem characteristics pertinent to the selection of 

engineering approaches must be identified. Twenty-three visual factors 
are listed below and discussed in terms of their criticality to design. 
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Scaling 
Resolution 
Detail Required 
Color 

Absolute B rightnes s 
Contrast Range 
Distortion Tolerance 
Flicker Tolerance 
Accommodation to Infinity- 
Relative Motion 
Independent Motion 
Coupled Motion 
Programed/ Unprogramed 
Angle of View 
Depth of Field 
Ease of Change 

Monocular Movement Parallax 
Inte r po s itioning 
Physical Size 
Power Consumption 
Original Hardware Cost 
Original Software Cost 
Maintenance Cost 

Scaling, This characteristic is concerned with the degree to which 
the medium used to store visual information is able to provide a simulated 
maneuvering area sufficient to meet the requirements without the necessity 
of changing scale factors. For example, for the purpose of low altitude 
visual navigation training, the inflexible scaling of the CCTV-model system 
is a severe disadvantage because of the conflicting requirements for high 
resolution and large area coverage. On the other hand, for a simulation 
problem such as ship-to-ship--underway replenishment involving the 
approach and station keeping of two large objects on an essentially flat, 
featureless and infinite plane (the open sea) this characteristic does not 
constitute a disadvantage for the CCTV system. 

Resolution, This is one of the main physical parameters of any 
system. For the purpose of this analysis, however, the specific values 
are less important than the gross categorization of systems in terms of 
whether their resolution is representative of optical quality, photographic 
quality or TV quality. 

Detail Requirements. This characteristic refers to the amount 
and type of detail which must be available in the visual scene to meet the 
mission requirements. It therefore defines the problems tb be solved in 
the selection and encoding of visual information for storage. As one might 
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expect, the importance of this characteristic varies widely depending on 
-the image storage medium involved. For example, it is relatively unim- 
portant in most photographically based systems, becomes a significant 
factor in the design of CCTV-model systems and is a critical aspect of 
computer-generated systems. 

Color Capability. This characteristic refers to the fidelity of 
color reproduction of the system. For the purpose of this analysis, only 
gross categorization is required. Systems can be classified simply on the 
basis of whether they are capable of producing black and white only, partial 
color as in systems based on 2-color mixtures, or full color as in systems 
based on 3-color mixtures or optically viewed models. 

Absolute Brightn3ss. This is a basic parameter of all simulation 
systems. For obvious reasons, low absolute brightness is frequently a 
serious disadvantage of systems designed to provide extremely wide fields 
of view. 

Contrast Range. This is a self-explanatory system characteristic 
In general, however, if a system delivers adequate brightness, the contrast 
range will also be adequate. 

r Jtortion Tolerance. In the context of this, analvsis, this charac- 
terise refers to the perceptual acceptability of simulation imagery which 
has been purposely distorted by the system, as in the VAMP-type semi- 
closed loop movies. 

Flicker Tolerance. This is a basic factor in both TV and motion 
picture systems. Although it does not usually present a problem, it can be 
a serious disadvantage in some applications of extremely wide angle motion 
picture systems. 

Accommo dation to Infinity. In the context of this analysis, this 
characteristic refers to the relative ease or difficulty of displaying the 
obtained imagery in such a way that it appears at infinity. For example, 
some systems lend themselves to the use of relatively compact virtual 
image displays, while others require that the image plane be physically 
remote from the viewer, therefore resulting in a large and expensive in- 
stallation to achieve the same effect. 

Relative Motion. This" factor is concerned with the angular velo- 
cities expected on the basis of mission requirements and with their antici- 
pated effects on various system parameters such as image smear, strobo- 
scope effects, and the detail requirements of the storage medium. 
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Independent Closed- Loop Motion . This characteristic refers to 
the relative ease or difficulty in providing independent closed-loop motion 
of targets within the simulated visual environment. For some missions 
such as air-to-air combat maneuvering! this capability is essential. A 
mission segment such as weapon delivery (including all types of missiles, 
fixed and flexible gunnery), which requires the provision of visual ground- 
effects /constitutes a special case in that the position of impact is deter- 
mined by ballistic computation (i.e. , the control loop is closed mechanically 
or electronically rather than manually). Simulation systems differ widely 
in their ability to provide this feature. For example , while it is possible 
to utilize a CCTV-model system for air combat simulation, .the solution is 
necessarily large and complex. Computer-generated displays, however, 
are well suited to such problems and, in addition, can provide any type of 
weapons effects desired, including even tracer paths. 

Coupled Motion. This factor is concerned with the feasibility of 
coupling a motion platform with the visual system under consideration to 
provide the integrated sensory cues required for vehicle control training. 
Mission considerations dictate both the requirement for, and the degree of, 
coupling. For example, for stable and slow-moving vehicles such as ships 
involved in harbor navigation and in docking, visual representation of the 
motion cues is sufficient in itself. Simulation of an assault boat landing, 
however, requires precise coupling of the two systems because the vehicle 
control task is critically dependent upon the relationship between the visual 
and motion cues . 



Closed- vs. Open- Loop Operations. This basic parameter is 
determined by whether or not control actions are fed back into the system. 
In open-loop systems, there is no control feedback; and the resultant < 
imagery is fixed or programed as in most vehicle control simulations or 
mechanically as in the simulation of weapons effects which was mentioned 
earlier. The critical importance of this parameter in many training situa- 
tions is due to the fact that closed- loop .ope ration demands active, partici- 
pation of the trainee in accomplishing the control task and provides feedback 
to him concerning degree of success. Special techniques are available to 
achieve these ends in certain types of open-loop simulation but, in general, 
they are more limited in their application and therefore less satisfactory. 



9 
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Angle of View. This is a critical parameter in the design of 
simulators for many training purposes. Other things being equal, it is 
desirable to provide the widest angle of view achievable. Some mission 
requirements, however, are more stringent than others in this respect. 
For example, for air-to-ground weapons delivery, a wide angle of view is 
desirable, but not essential, since bomb, missile and fixed gunnery targets 
will generally be close to the flight path. On the other hand, for simulation 
of the low altitude navigation problem, visual information from the extreme 
periphery is essential and therefore, an extremely wide angle display is 
required. 
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Depth of Field. This is a basic parameter which does not usually 
constitute a problem for most system/mission combinations, since in most 
cases, the actual distance from optical probe to area of interest is suffi- 
ciently great that a normal optical system stopped dcwn to a small aperture 
will yield adequate depth. A very small lens aperture, of course, creates 
other problems such as the necessity to either increase the illumination 
level or the sensitivity of the recording device. There are, however, 
mission specific situations in which depth of field does present a serious 
system limitation independent of both brightness and sensitivity, e.g. , when 
an optical probe is required to come very close to a surface such as in a 
CCTV-model simulation of carrier takeoff and/or landing. 

Ease of Change. This factor refers to the relative ease of making 
changes in the visual information which is already coded and stored in the 
system. It is, of course, related to, but not identical with, the scaling 
factor discussed earlier. While flexible scaling refers to the complete re- 
placement of one stored visual environment by. another having a different 
scale factor, ease of change refers to the relative simplicity of updating 
an existing image store by making small localized changes. This charac- 
teristic can be illustrated by comparing the photographic (factored transpar- 
ency) and the computer-generated approaches to radar landmass simulation. 
Factored transparency systems which represent geographic and cultural 
features as a function of elevation in one transparency and as a function of 
radar reflectivity in another, cannot.be updated when,, for example, radar 
targets such as bridges, highways, dams, reservoirs, etc., are created 
or destroyed. It is, of course, apparent that computer-generated imagery 
can be updated with ease by appropriate programing changes . 

Monocular Movement Parallax. This is one of several factors con- 
cerned with the cues which govern the perception of apparent depth in the 
imagery delivered by a system. Monocular movement parallax refers to 
whether or not objects in the background of a scene appear to move in rela- 
tion to the foreground, as a function of relative motion between the observer 
and the target area. Since both the direction and velocity of this apparent 
motion constitute cues to the depth of an object in the real world, this factor 
is a powerful determiner of the corresponding illusion in the simulated en- 
vironment. Generally, monocular movement parallax is based upon three- 
dimensional inputs, since it depends upon constantly changing aspect angles 
between the recording device and objects in the visual field. Although a 
regularly changing target aspect angle is a necessary condition for deriving 
the movement parallax depth cue, it is not in itself sufficient; the target 
aspect angles must be changing at or above some threshold value. The 
practical significance of these considerations is that, taken together, they 
define a range limitation for some missions (e.g. , air-to-ground target 
acquisition, remotely controlled (TV) guided missiles) beyond which depth 
cues based on this principle will not be effective. 
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Interposition, This , like monocular movement parallax, is another 
of the basic cues to depth. However, unlike the former, the illusion of 
depth is not dependent upon motion, but upon the individual's learned re- 
sponse to certain characteristics of identifiab.le patterns. Through experi- 
ence, an individual learns to interpret certain shapes as representative of 
certain classes of objects . When the expected image of a man interrupts 
or cuts off a portion of the image to be expected of a tree, the man is in- 
terpreted as being in front of (closer to the observer than) the tree. Pro- 
vided that the geometric relationships between the relative size of known 
objects is preserved, within very broad limits, interposition is a simple 
but very powerful cue to depth. Most simulation systems which are based 
upon three-dimensional inputs (optical or TV-viewed scale model, movies 
based on either the real world or models if it) provide this cue automatically. 
Artificially calculated visual environments (e.g., computer-generated 
imagery, animated motion pictures) can also provide these cues, but at 
some cost in programing effort. In some cases, however (e.g. , point light 
source - transparency systems), the inability to provide this cue is a sub- 
stantial disadvantage of the system. 

Physical Size This is an obviously important aspect of any system 
in that it is closely related to overall cost. In the context of our analysis, 
only gross categorization is necessary. These categories: are defined as 
follows: 1) small-desk-top to small classroom size, 2) medium-a space 
requiring special preparation such as an unusually high ceiling, a false 
floor, special cooling arrangements, etc., and 3) large- requiring a special 
building, exceptionally large room, or an unusually extensive modification 
Ox an existing building. 

both rpfi ^r* ^ C ° nSUm P tioi V This is mother cost-related parameter which 
both reflects and is reflected in the size, weight, illumination and cooling 
requirements of the system. For the purpose of this analysis, the following 
categories are defined: • 5 

1) small- -up to 10 KW 

2) medium- -10 to 50 KW 

3) large--over 50 KW 

, „ . ° ri g inal Hardware Cost. For the purpose of this analysis the 
following categories are defined: 

Low- -up to 250K 
Medium- -250K to 1.5M 
High— above 1 . 5M 

- O^al Software Cost, For the purpose of this analysis, original 

software costs are classified^ follows: • 
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Low 

Medium 

High 

Maintenance Costs. This factor includes both hardware (parts re- 
placement and equipment repair) and software (programing) support required 
for adequate availability. For the purpose of this analysis maintenance costs 
are classified as follows: 

Low 

Medium 

High 

Some Insights About Visual System Characteristics . It is apparent 
that the importance to be attached to any individual system characteristic 
varies depending on both the mission and the simulation system involved. 

While all of the 23 characteristics cited are relevant to some com- 
bination of mission segment and simulation technology, few, if any,. have 
relevance for the entire range of such possible combinations. 

A given system parameter, such as resolution (which is a quanti- 
tatively measurable aspect of any display system), is a significant charac- 
teristic only if it is substantially greater than, or less than that required 
on the basis of mission considerations. For example, if the available 
system resolution exceeds the resolution required to simulate a given op- 
erational environment, the proposed system represents a significantly 
"overengineered" (and therefore overpriced) solution in respect to this 
particular parameter. Similarly, wherever the system capability is sub- 
stantially less than the requirements, the parameter in question constitutes 
a significant disadvantage of the system for that specific application. 

While only those systems characteristics for which requirements 
and capabilities are for some reason unequal are to be considered signifi- 
cant, it is apparent that not all significant characteristics are important 
from the practical standpoint. This is because the nature of our primary 
interest necessarily emphasizes the disadvantages or shortcomings of com- 
peting technologies rather than their advantages. For example, in our 
context, a parameter which is identified as a system advantage for a given 
application usually requires no further consideration. Conversely, a char- 
acteristic identified as a system disadvantage must be further considered 
in terms of its criticality, i.e. , a system disadvantage may be either a 
critical or a non-critical factor in the ultimate hardware selection process. 
The designation of individual system characteristics as important or critical 
in achieving a specified simulation goal is a judgment which requires con- 
siderable knowledge and experience on the part of the specialists involved.. 
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These judgments constitute an important step in the overall decision process 
in that they identify those difficulties which must be overcome by each of the 
candidate technologies in order to achieve the desired objective. In terms 
of this analysis, a critical system deficiency for any specified application 
is defined as one which can be circumvented only at considerable cost to 
the system, in terms of money, complexity, developmert time, maintain- 
ability and engineering risk. 

Step 4; Construct a series of matrices correlating visual 
simulation requirements with system characteristics 

The application of this technique to an actual visual simulation 
problem is illustrated with an analysis of three candidate simulation tech- 
nologies as a function of the same set of simulation requirements. These 
are: 1) CCTV, 2) open-loop movie, and 3) computer -generated simulation 
systems. The low altitude visual navigation problem (see Table 3) is selec- 
ted because of the stringent requirements it imposes on the hardware solu- 
tion with respect to several important system characteristics, viz. , angle 
of view, scaling, resolution, depth cues, and the desirability of closed- 
loop operation. The analysis will result in three separate system charac- 
teristic profiles on the basis of which the relative strengths and weaknesses 
of each of the candidate systems may be compared. 

The format adopted for the multiple decision matrices is illustrated 
in Figure 3. With few exceptions, which are noted below, all matrices are 
identical, consisting of nine cells in a 3x3 block arrangement. There is an 
individual matrix for each of the 23 system characteristics (e.g. , scaling, 
resolution) previously discussed. Each of these general system character- 
istics is evaluated in terms of the mission-derived simulation requirements 
on the vertical axis, and the specific system characteristics available on 
the horizontal axis. Each cell of the matrix is pre-numbered with a "value" 
ranging from -2 to +2 (the specific digits in themselves have no quantitative 
significance). The primary feature of each matrix is that the digits are 
arranged so that a "0" score is obtained whenever, in the judgment of the 
specialist, the simulation requirement as determined by the nature of the 
mission is met by the specific characteristic of the system under consider- 
ation. Matrix cells containing negative numbers will be obtained when the 
judgment is made that the specific system characteristic is not adequate to 
meet the requirement. Similarly, a positively numbered cell indicates an 
unneeded or excess capability of the system in question for the purpose of 
achieving the defined objective. It should be noted that, since the format 
of the matrices is essentially uniform throughout, the specific judgments 
required for each of the requirement/capability combinations vary some- 
what. For example, some matrices are simply labeled high, medium and 
low on each of the axes. In these cases, which tend to be associated with 
basic system parameters, the judgments to be made are, a) what level of 
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TYPE OF VEHICLE & GENERAL MISSION: 

SPECIFIC MISSION SEGMENT: 

SYSTEM TYPE 



VISUAL SIMULATION 
REQUIREMENTS 



SYSTEM CHARACTERISTICS 



Flexible Scaling 



Brightness 



Resolution 



Color 



Contrast Range 



Primary 


Secondary 


Primary 


Advantage 


Advantage ] 


Disadvantage 


Required 


0 


-1 


-2 


Desired . 


1 


0 


-1 


Not Necessary 


2 


1 


0 


High 


Medium 


Low . 


High 


0 


-1 


-2 


Medium 


1 


0 


-1 


Low 


2 


1 


0 




High 


Medium 


Low 


High 


0 


-1 


-2 


Medium 


1 


0 


-1 


Low 


2 


1 


0 




Optical 


Photo 


TV 


Optical 


0 


-1 


-2 


Photographic 


1 


0 


A 


TV 


2 


1 


0 


Full 


Partial 


Absent 


Full 


0 


-1 


-2 


Partial 


1 


0 


-1 


Absent 


2 


1 


0 




High 


Medium 


Low 


High 


0 


-1 


-2 


Medium 


1 


0 


-1 


Low 


2 


1 


0 




Inherent Distortion 




Low 


Medium 


, High . 


Low 


0 


-1 


-2 


e)Medium 


1 


0 


-1 


High 


2 


1 


o 



Figure 3. Format for the Multiple Decision Matrices (Part 1 of 3) 
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VISUAL SIMULATION 
REQUIREMENTS 



SYSTEM CHARACTERISTICS 



Primary Secondary Primary 



Accommodation to 


Required 


0 


-1 


-2 


Infinity 


Desirable 


1 


0 


-1 




Unnecessary 


2 


1 


0 1 



Primary Secondary Primary 



Independent Motion Required 
of Targets in Field Desirable 

Unnecessary 



0 


-1 


-2 


1 


. 0 


-1 


2 


1 


0 



Flicker (tolerance ) 



(closed loop) 



None 

Medium 

High 



Unprgmd 



Semi-Prgd Prgrmd 



Control of Visual Unprogramed 


0 


-1 


-2 


Environment Semi-Prgmd 


1 


0 


-1 


Programed 


2 


1 


0 



Degree of Flicker 



0 


-1 


-2 


1 


0 


-1 


2 


1 


0 



Low' 



Relative Motion 
of Targets 



High <Vlcty 
Medium <^ Vic 
L»ow^ Vlcty 



Susceptability to Smear 

Medium High 



0 



-1 



-2 



-1 



Difficulty/Expense of Providing 







Low 


Required Coupling 

Medium High 


Coupling of Visual 


Required 


0 


-1 


-2 


Field with Real 


Desirable 


1 


0 


-1 


Motion 


Unnecessary 


2 


1 


o 




Wide 




Medium 


Narrow 




Wide 
(140° up) 


0 


-I 


-2 


Angle of View 


Medium 


1 








( 90° -140°) 


0 


-I 




Narrow 






1 






(less than 90°) 


2 




0 



Figure 3 . Format for the Multiple Decision Matrices (Part 2 of 3) 
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VISUAL SIMULATION 
REQUIREMENTS 



SYSTEM CHARACTERISTICS 



( 







Extensive 


Moderate 


Small 


Depth of Field 


Extensive 


0 


-1 


-2 


Moderate 


1 


o 


-1 




Small 


2 


1 


0 


Monocular 


Required 


Primary- 
Advantage 


Secondary 
Advantage 


Primary 
Disadvantage 


0 


-1 


-2 


Movement Parallax 


Desirable 


1 


0 


-1 




Unnecessary- 


2 


I 


0 


Ease of Changing 




^ 1 


Medium 


Low 


High j 




-1 


-2 


Visual Environment 


Medium 


1 


0 


-1 




Low 1 


2 


1 


0 


Interposition 
(Terrain Masking) 




Primary 
Advantage 


Secondary 
Advantage 


Primary 
Disadvantage 


Required 


0 


-1 


-2 


Desirable 


1 


0 


-1 




Unnecessary 


2 


1 


0 


Size of Installation 


Small 


Medium 


Large 


1 


0 { 


-1 1 


o 1 




.Low 


Medium 


High 


Original Hardware Cost 


0 j 


• -1 


I 2 . ] 


Maintenance Cost 


Low 


Medium 


High 


l 


0 1 


'-I I 


-2 


Power Consumption 


Low 


Medium 


High. 


1 


0 j 


-1 


- 2 1 


Low 


Medium 


High 




0 | 


-* 1 


-i 



Figure 3. Format for the Multiple Decision Matrices (Part 3 oS 3) 
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the specific characteristic is required on the basis of the known facts about 
the mission, and b) what is the gross ability of the system in question rela- 
tive to the specified characteristic. In some cases, the same basic judg- 
ments are required, but for convenience, the axes are labeled in terms of 
the characteristic itself. Thus, angle of view, is labeled in terms of wide, 
medium and narrow; color, in terms of full, partial or absent. 

Another type of judgment is required for such system character- 
istics as independent motion, flexible scaling, accommodation to infinity, 
monocular movement parallax and interposition. In these cases, the perti- 
nent judgments are: a) whether the characteristic in question is required, 
desirable or unnecessary in order to achieve the simulation objective as 
defined by the mission, and b) whether the characteristic constitutes a pri- 
mary advantage, a secondary advantage, or a primary disadvantage of the 
system under consideration. 

Finally, with respect to several cost- related factors such as power 
consumption, maintenance costs, etc. , the usual nine cell matrix is reduced 
to three. This is because by definition, only low cost, low maintenance, 
low power consumption, etc. , is required (i.e. , desired). 

NOTE: The steps described thus far are accomplished 
by the human factors specialist* Beginning with Step 5, 
the output reflects the combined efforts of human factors 
and engineering working interactively. 

Step 5: Develop a system characteristic profile for each candidate 
engineering approach depicting the matrix scores for each visual 
system characteristic 

The format of the system characteristic profiles derived from the 
multiple matrix scores is illustrated in Figure 4 # The 23-system charac- 
teristics selected for analysis are arrayed along the horizontal axis and 
individual matrix scores are plotted vertically. When the individual points 
are connected, a "profile 11 is formed which graphically portrays the ability 
of the simulation system under consideration to meet the simulation require- 
ments of the selected mission. A given profile is concerned with the "fit" 
of only one system/ mission combination; successive analyses are made for 
each of several candidate systems in order to provide a basis for comparing 
alternative engineering solutions. The development of the profiles is 
accomplished by human factors and engineering specialists in consort to 
insure that competitive technologies are most systematically and logically 
evaluated* This is an important requirement since the precise definition 
of all the visual cues necessary to meet the training requirement for a given 
mission is often such a difficult, and time-consuming task that it constitute? 
a major disadvantage of the approach in itself. 
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TYPE OF VEHICLE & GENERAL MISSION: 

SPECIFIC MISSION SEGMENT: 

E VALUATOR (S): : _ 

SIMULATION SYSTEM: 

CRITICAL FACTORS MATRIX SCORE 

-2 -1 0 4-1 + Z 

Scaling (3) — H 

Resolution — _ 

Detail Required — 

Color : 

Absolute Brightness 

Contrast Range , 

Distortion Tolerance 

Flicker Tolerance ■ . 

Accomodation to Infinity — 

Relative Motion 

Independent Motion 

Coupled Motion — 

Programmed/Unprogrammed(2) : 

Angle of View (1) — 

Depth Of Field 

Ease of Change . 

MonocularMovement Parallax - 

Interpositioning 

Physical Size "■ 

Power Consumption 

Original Hardware Cost 

Original Software Cost 

Maintenance Cost 

Figure 4. Format of System Characteristics Profile 
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The results of the matrix analyses of the three candidate simula- 
tion systems with respect to the simulation requirements of the low altitude 
visual navigation mission are summarized next. For the low altitude visual 
navigation problem three system characteristics were designated as critical 
factors. These are, in the order of their presumed importance, angle of 
view, programed vs. unprogramed, and scaling. 

In order to fully understand the derivation and use of the informa- 
tion presented in the matrix profiles , it is first necessary to understand the 
visual simulation requirements of the low altitude, visual navigation mission. 
The simulation requirements of this mission indicate that: 

a> An extremely wide angle of view is essential since many of the 
visual cues necessary for proper geographic orientation at low altitude are 
derived from the peripheral field. 

b. An unprogramed (closed-loop) capability is highly desirable 
since the trainee must be able to exercise active control of his flight path* 
Whether or not this feature is essential (as opposed to desirable) is a 
matter of interpretation. It can be argued that since the navigation problem 
involves only minor deviations from a prescribed path down a defined series 
of terrain corridors, only a semi-programed capability is required. 

c. Some form of flexible scaling is essential because the nature 

of the low altitude navigation mission requires large area coverage at fairly 
high resolution* 

d. The ability to provide adequate depth cues is essential because 
the shallow look down angles, relatively short slant ranges, and high angu- 
lar velocities characteristic of low altitude flight produce many operation- 
ally important cues which are based on interposition and on monocular 
movement parallax (e.g. , target/ terrain masking -unmasking). In this 
particular analysis, however, neither monocular movement parallax nor 
interposition were designated as critical factors because all three candidate 
systems are capable of meeting the requirement. Thus, although these 
characteristics are essential to a successful simulation of the low altitude 
navigation mission, it would serve no purpose to designate them as critical 
factors since they would not differentiate between possible engineering 
solutions* 

In addition to the 'above, it should be noted that the requirement 
for visual information in the extreme peripheral field has implications for 
several other system characteristics. Thus, in an extremely wide angle 
display, the absolute brightness, flicker tolerance, accommodation to 
infinity and relative motion characteristics take on added importance and, 
in fact, can become critical. 
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Figures 5, 6 and 7 depict system characteristics profiles for the 
three candidate simulation systems. A comparison of the three profiles 
reveals that for the simulation of the low altitude visual navigation mission: 

a. The CC TV-model system has two major disadvantages, viz, , 
narrow angle of view and difficult scaling problems. In addition, it requires 
a relatively large and costly installation. The primary advantage of this 
type of approach is in its ability to provide unprogramed, or closed-loop 
operation. The profile also, shows that resolution is marginal and there is 
an appreciable susceptibility to image smear. On the other hand, color, 
brightness, contrast, distortion, depth cues and original costs are consi- 
dered adequate. 

b. The wide angle open-loop movie system has (as its name implies) 
one major disadvantage, open-loop operation, and one major advantage, the 
ability to provide extremely wide angle imagery. Flexible scaling which 
presents a major problem for the CCTV-based system is an inherent prop- 
erty of the motion picture -based system. Resolution, color, distortion 

and depth cues were all judged to be adequate. There are, however, some 
problems with stroboscopic effects and image smear in the extreme peri- 
phery. Size, hardware and maintenance costs are more or less moderate, 
but the cost of acquiring properly controlled imagery (software) is high. It 
should be noted here that care must be exercised in the interpretation of any 
matrix score shown on a profile, since any given score is the result of the 
interaction of simulation requirements with system characteristics. Thus, 
in Figure 5, both the independent motion characteristic and the ease of" 
change characteristic are plotted as "O's," meaning that the simulation re- 
quirement is met by the specific system characteristic in question. This 
is in fact true, for both cases . However, the key to the effective use of this 
information is in the mission-derived simulation requirements. For ex- 
ample (refer to Figure 3), independent motion of targets within the visual 
field is not considered a necessary feature of the low altitude visual naviga- 
tion problem. Even though the inability of the open-loop movie system to 
provide such motion is a primary disadvantage of the system per se, it does 
not constitute a disadvantage in achieving the simulation objectives of this 
mission. Similarly, because the ability to make small localized changes 
in the stored imagery is not required in this case, there is no discrepancy 
between the simulation requirements and system capability concerning the 
ease of change characteristic. 

c. The computer-based imagery system is certainly the most 
flexible and therefore potentially the most useful of all the simulation tech- 
nologies. However, its extreme flexibility is both its greatest strength and 
its greatest weakness. As mentioned previously, one of the greatest ad- 
vantages of computer-generated imagery is that the available resolution 
elements ( "edges M or M lines ,f depending on the specific technique involved) 
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TYPE OF VEHICLE & GENERAL MISSION'S Fixed- Wing - Air-to-Ground 

SPECIFIC MISSION SEGMENT: Low Altitude Visual Navigation 

E VALUATOR (S): 



Human Factors Specialist 



SIMULATION SYSTEM: 



CC TV-Model 



CRITICAL FACTORS 



-2 



MATRIX SCORE 
-1 0 4-1 




Scaling (3) 
Resolution 
Detail Required 
Color 

Absolute Brightness 
Contrast Range 
Distortion Tolerance 
Flicker Tolerance 
Accomodation to Infinity- 
Relative Motion 
Independent Motion 
Coupled Motion 

Programmed/Unprogrammed(2) 

Angle of View (1) 

Depth Of Field 

Ease of Change 

MonocularMovement Parallax 

Interpositioning 

Physical Size 

Power Consumption 

Original Hardware Cost 

Original Software Cost 

Maintenance Cost 



Figure 5. System Characteristics Profile for the CCTV-Model 
Simulation System. 
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TYPE OF VEHICLE & GENERAL MISSION : Fixed Wing - Air to Ground 
SPECIFIC MISSION SEGMENT: Low Altitude. Visual Navigation 

EV ALU ATOR (S ) : Human Factors Specialist 

SIMULATION SYSTEM! Open Loop Movie 



CRITICAL FACTORS . 

Scaling {3) 
Resolution 
Detail Required 
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Absolute Brightness 
Contrast Range 
Distortion Toletance 
Flicker Tolerance 
Accomodation to Infinity 
Relative Motion 
Independent Motion 
Coupled Motion 

Programmed/Unprogrammed(2) 

Angle of View (1) 

Depth Of Field 

Ease of Change 

Monocular Movement Parallax 

Inter positioning 

Physical Size 

Power Consumption 

Original Hardware Cost 

Original Software Cost 

Maintenance Cost 
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Figure 6. System Characteristic Profile for the Open Loop 
Movie Simulation System. 
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TYPE OF VEHICLE «c GENERAL MISSION: Fixed Win g j Air to GrQund 

SPECIFIC MISSION SEGMENT: Low Altitude Visual Navigation 

EV ALUATOR (S ) : Human Factors Specialist , 

SIMULATION SYSTEM: Computer Generated 



CRITICAL FACTORS 



-2 



MATRIX SCORE 
•1 0 4-1 +2 




Scaling (3) 
Resolution 
Detail Required 
Color 

Absolute Brightness 
Contrast Range 
Distortion Tolerance 
Flicker Tolerance 
Accomodation to Infinity 
Relative Motion 
Independent Motion 
Coupled Motion 

Programmed/Unprogrammed(2) 
Angle of View (1) 
Depth Of Field 
Ease of Change 
MonocularMovement Parallax 
Inter pos itioning 
Physical Size 
Power Consumption 
Original Hardware Cost 
Original Software Cost 
Maintenance Cost 



Figure 7. System Characteristics Profile for the Computer 
Generated Simulation System 
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can be differentially apportioned according to the areas of interest in the 
visual scene. For example, in an air-to-air combat simulator, most of 
the available resolution elements could be devoted to defining the image of 
other aircraft in the visual field at the expense of ground detail, which is 
only needed for gross orientation. While this capability is a distinct advan- 
tage of the computed image technique, implementing it presents a major 
problem This is directly expressed in the detail requirement character- 
istic, and indirectly reflected in the high software cost factor. Of the sys- 
tem characteristics which were previously designated as critical factors, 
the computer-generated technique is deficient in only one, i.e. , angle of 
view. However, the highly negative detail requirement factor should be 
regarded as a special disadvantage of this technique because of its influ- 
ence on the values of the closely related scaling, resolution, color, depth 
and cost factors. Obviously, in any system based on the manipulation of 
a mathematically defined visual environment each element of that environ- 
ment must be carefully preselected and incorporated with all of its arbi- 
trarily assigned parameters (such as color) into the mathematical model. 

Beyond the profile analysis, it is recommended that the team de- 
velop a quantified figure-of-merit for each candidate engineering solution 
for meeting the design requirements. This requires that careful thought 
be given to differentially weighting the characteristics in the profile (with 
an identification of the critical requirements which must be satisfied) An 
attempt to quantify (via FOMs) serves two purposesTTTenables greater 
precision in arguing the relative merits of candidate approaches; and con- 
tributes towards improvements in the technique by the "forced" develop- 
ment of empirical weightings which, with experience, may eventually be 
established for each system characteristic. 

Summary 

The analytic technique described here provides a means for organizing 
displaying, and evaluating the technical information which is already avail- 
able concerning the interaction of visual simulation requirements and tech- 
niques. While the basic approach employed is imperfect, it is believed to 
be sound and logically defensible and worthy of refinement by practical use 
Figure 8 summarizes the series of steps accomplished in the procedure. 

2.3.3 CASE 3: Technique for Defining Display and Control 

Requirements in Generalized Trainee Station Design 

There are instances where the purpose of training is best achieved 
via a generalized training device (either single station or multiple but in- 
dependent stations). In this situation, device design is general to a variety 
of vehicles, but not identical in layout, dimensions, equipment, etc., to 
any existing vehicle; in addition, an array of displays and controls provides 
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only that information needed by the student to perform the functions of the 
job. The selection of the instruments needed to provide these vital functions 
and the kinds and amounts of information displayed are the issues of concern. 
Implicit in this situation is a substantial simplification in representation 
while maintaining behavioral similarities between the device and the opera- 
tional system jobs. An emphasis is placed on simulating the stimuli and 
responses found in the operational situation. The human factors effort is 
to specify the information requirements needed for this type of training 
wherein the content is relevant to a variety of operational situations. The 
design rationale is to provide only those functions needed in job perform- 
ance. This involves the installation of a task environment which affords the 
student the opportunity to learn and retain skills which are common to a 
number of systems. 

A technique is provided here for deriving information requirements 
for situations calling for generalized displays and controls wherein 1) the 
stimuli and responses of the operational environment are simulated (i.e. , 
behavioral similarities between the device and the operational counterpart 
functions), and 2) simplification in the degree of fidelity of representation 
is achieved commensurate with the generalized functions provided. In this 
case, the content deals only with the design requirements for the displays 
and the associated controls for enhancing the instructional process. That 
there are other human factors requirements involved with training devices 
of this type is tacitly assumed. 

The procedure outlined below utilizes Device 1D23 as the model for 
the analysis 1 . Device 1D23 is a multi-station Air Navigation Trainer (forty 
independent cockpits, six instructor stations) which simulates characteristic 
navigator functions generalizable to a variety of aircraft types although the 
cockpits are not identical to any given aircraft. The purpose of the trainer 
is to instruct Naval Flight Officers in basic air navigation and communica- 
tion procedures and concepts. The device characteristics include: 

• preprogramed lesson plans with enroute modification capability 

• computer-controlled cockpit environment 

• automated measurement capability 



The analytical derivations are based on two sources: 1) Specification for 
Trainer, Communication and Navigation, Device 1D23, 313-1115, Task 
1309i U.S. Naval Training Device Center, Orlando, Florida, 5 January 
1971, and 2) Bark, M. , et al, Design Study Report for a Multiple Trainer 
Station Air Navigation Trainer, NAVTRADEVCEN 68-C-0308-1, U.S. 
Naval Training Device Center, Orlando, Florida, January 1969. 
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. instructional assists in the cockpit (annunciator panel, 
performance status indications, knowledge of performance 
and cueing information) 

. communications between student and instructor/operator 
• student flight command controls 

. student response controls to respond to instructor or to 
computer interrogations. 

In this case example, the emphasis is placed on deriving informa- 
lon requirements relative to the design of generalized displays and controls 
for a training device. It should be made explicit that no consideration is 
given here to other legitimate concerns that must be accounted for in trainer 
development such as actual design of instruments, cockpit panel layouts, 
communication networks, response keyboard design as well as other human 
factors items in the technical approach documentation. The procedure is 
concerned only with deriving the stimulus and response capabilities neces- 
sary for the student to learn the generalized navigation functions, and cen- 
ters on: the stimulus requirements for operational instruments; the display 
and control of specific stimuli, and the command and response capability 
provided the student. The issues to consider in assembling the information 
requirements pertinent to engineering design are outlined next. 

.°P erat f ona l Instruments . The major group of instruments are 
it 086 as8ociated the direction and movement of the aircraft in flight 
These instruments provide the specific stimuli for the learned navigation' 
responses and are of most concern in station design. The approach to de- 

™^ff- infor , mation requirements that must be represented in this instru- 
mentation should consider the following. 

a. Provide the student only that information needed to effectively 
perform navigation problems (in a form compatible with processing require- 
ments). The content of instruction for the navigation tasks involves: 

• basic dead- reckoning navigation 

• basic airways navigation 

• fuel management 

• relative motion training 

b. From an examination of each of the content tasks, the instru- 
ments necessary to perform navigation functions are identified. 

c From this analysis, a listing.of instrument stimuli is derived. 
Displays, indicators, and panels are to be utilized in a computer-controlled 
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cockpit environment, hence many of the standard operational instruments 
are inapplicable or impractical. The displays and indicators selected for 
simulation are primarily those associated with navigation problem solutions. 
Additionally, a limited number of supporting instruments will give the stu- 
dent aircraft attitude and rate information. The following requirements 
were derived for instrument displays. 

Instruments Required for Navigation Problem Solutions: 

Indicated Airspeed 
Mach No* 
Altimeter 
Magnetic Compass 
Radar Altimeter 

Bearing Distance Heading Indicator (BDHI) 
Eight Day Clock 
Outside Air Temperature 
Fuel Flow Gage 
Fuel Quantity Gage 

Instruments Depicting Attitude or Motion, Requiring No 
Dependency for Navigation Problem Solution: 

Gyro Horizon or Attitude Gyro 
Rate of Climb 

Communication and Navigation Displays and Controls: 

Communication Navigation Display Panel 
Communication Navigation Control Panel 
Navigation Control Panel 
Navigation Display Panel 
MA-1 Compass Control Panel 

d. Actual aircraft instruments will not be employed. Digital 
displays are desired for all instruments relevant to the navigational prob- 
lem. Lighted numeric readouts feature faster response, higher stability 
and finer resolution and accuracy than the needle type display (also, the 
computer interface to lighted numeric readouts requires the simplest logic 
of any simulation technique). 

e. The specific information to be displayed on each selected 
instrument must be defined. 

f . To display rate or relative position information, hybrid in- 
strumentation is advisable (combined digital and analog techniques). The 
following instrument components are simulated by the use of rate servos: 
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• gyro heading card (Bearing Distance Heading Indicator) 

• ADF needle (BDHI) 

• TACAN/VOR needle (BDHI) 

• Magnetic heading card (wet compass) 

• Pitch and roll (gyro horizon) 

g. The response characteristics of each instrument must be 
determined. A simple approach for achieving this is to evaluate response 
across a range of update rates. In our example, two types of instrument 
responses were evaluated based on update rates ranging from twice per 
second to once per four seconds, presented to subjects who rated the 
"comfortableness 11 of the movements for each rate. Based on this concen- 
sus, an update rate of once per second was recommended for both digital 
movements and for needle movements. 

h. A functional layout scheme must be formulated for each of 
the instruments selected for use in the cockpits, based on importance and 
frequency of usage. In the 1D23 instance, estimates were made by subject- 
matter experts on the importance of the instruments in navigation. Each 
instrument was rated on a scale of 1 to 10 in terms of increasing criticality 
or importance to navigation. The average ratings were as follows: 



Instrument 


Rating 


1) BDHI 


1.2 


2) Radar (Blank) 


1.4 


3) Indicated Airspeed & Mach No. 


1.8 


4) Pressure Altimeter 


2.0 


5) CNI Panel 


3.2 


6) Navigation Display- 


3.0 



• Doppler NAV 

• TAS 



7) Radar Altimeter 3.8 

8) Fuel Flow 4. 1 

9) Fuel Quantity 4.6 

10) Outside Air Temp 5.2 

11) Gyro Horizon 5.6 

12) Rate of Climb 5.4 

13) 8 -Day Clock 5.6 

14) Mag Compass 5.0 



2.3.3.2 Specific Stimuli Presentation. Specific stimuli associated with 
critical and complex sequences of performance are not easily identified. 
This presents problems for instructional control in complex system opera- 
tions, particularly in multi-station trainers where instructor personnel 
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are not able to monitor individual performances continuously and must 
necessarily time -share among students in the development of instructional 
strategies. Thus, specific stimulus inputs requiring observable student 
performance in the presence of the stimulus are desirable as an assist to 
instructional control (behavior modification). An annunciator panel was 
recommended for Device 1D23 as a means for providing this precise stimu- 
lus control. This type of display presents questions at programed times 
which call for specific behaviors by the student. The response to these 
specific stimuli is monitored by the instructor on a CRT. A series of short 
programed questions are individually presented, requesting certain infor- 
mation from the student. Each question is programed to be visually dis- 
played at specified times in the mission cycle and remains "on" for at least 
30 seconds during which latency of response is recorded up to 2 minutes. 
Failure of the student to respond (via keyboard entry) to the question within 
the programed limit is recorded as an incorrect response. In all cases, 
information feedback is given to the student, via the computer, immediately 
upon completion of response to each message. A record of these transac- 
tions is stored in mass .memory for post-mission critique. Potential annun- 
ciator questions, developed in conjunction with the lesson plans include the 
following: 

• CALC ETA to Next Fly- To Point 

• CALC Bingo Fuel 

• CALC Bingo Time 

• CALC TB to Point ZULU 

. CALC MB to Next Fly- To Point 

• CALC WS 

• DME TACAN Theta 

• CALC TAS 

• CALC Pres. Posn. 

• CALC Dens. Alt. 
. CALC VH 

• CALC True GS 

• CALC Intercept Time 

Each message, containing a maximum of 30 characters, will 
be used to determine the student's knowledge during inflight procedures. 
The trainee station computer will monitor and store both the question and 
response, providing a recorded means for critiqueing performance. The 
student responses to the annunciator questions are entered via the keyboard 
which provides the response options shown in Table 5. 

2.3.3.3 Command and Response Capability. Means must be provided 
the student for giving commands to the "pilot" (vehicle control) and for 
responding to instructional interrogation. All inputs for pilot commands 
and for instructional responses are entered and transmitted by keyboard 



-61- 



NAVTK ADEVCEN ? i-C-00 1 i - 1 



TABLE 5. TRAINEE RESPONSE PUSHBUTTON FUNCTIONS 



No 


x'USixDUttOn 


Function 


Units 


1 


a iiuc * rxT / ivi in 


v^aicuiatea x lme 


Hour 8 / minutes 


2* 
**• 


opeea — x\nots 


Calculated Airspeeds 
or Ground Speed 


Knot 8 


3. 


B earing /Heading 


Calculated Bearing 
or Heading 


0-360° 


4. 


Fuel - Ibs/pph 


. Calculated Fuel 
Quantities or Fuel 
Flow 


pounds or pounds 
per hour's 


5. 


Distance - N # M. 


Calculated Distances 


Nautical miles 


6. 


Altitude - Feet 


Calculated Altitudes 


Feet 


7. 


Latitude - Deg/Min 


Calculated Latitude 


Degree s /minutes 


8. 


Longitude - Deg/Min 


Calculated Longitude 


Degree s / minutes 



ERIC 
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and monitored each half-second by the trainee station computer* Since no 
pilot is involved in the navigation problem, the student recommends flight 
regimes to the computer by selecting one of eight available options, shown 
in Table 6* The command and response equipment also provides perform- 
ance information to the student* A status indicator displays the results of 
the student responses* When an answer is correct or in-tolerance, the 
word, "correct, " is illuminated; when an answer is incorrect or out-of- 
tolerance, the word, "incorrect," is illuminated together with a two-digit 
display indicating extent of error* 

Stimulus support is available to the student via a position re- 
quest > control and indicator* The student is afforded the option of request- 
ing position information (latitude and longitude) when he suspects that his 
calculations are in error or that he is lost* 

2*3*4 CASE 4: Employing Training Technology to Enhance 
Training 

The cases provided thus far have been concerned with the informa- 
tion requirements relative to variations in fidelity on the engineering con- 
tinuum* Several levels of fidelity are appropriate in installing the required 
instructional environment in the trainee station* High fidelity physical cor- 
respondence to the operational environment is desired in situations demand- 
ing direct transfer of training from the device to the operational system, 
particularly in advanced instruction and in training involving emergency 
conditions* Deliberately backing off from duplicating the characteristics 
of the operational situation (e*g* , reductions in engineering tolerances, 
generalized training devices, part-task trainers, procedures/familiariza- 
tion trainers, etc*) is also a prevalent design option* The salient feature 
here is the cost economy achievable due to reductions in engineering fidelity 
without compromising training effectiveness* In design areas where the 
engineering state-of-the-art is less than adequate, considerable attention 
must be devoted to insuring that the unintentional reductions in tolerances 
are the best achievable. Instructional advantage is forfeited when the re- 
sultant attenuations in fidelity are not perceived as valid by the student* 

Case 4 exemplifies still another approach to simulation fidelity in 
trainee station design, involving planned deviations in configuration or func- 
tion associated with the operational system counterpart* This differs from 
a cost-effective lowering of the engineering fidelity or a reduction in fidelity 
tolerances where the engineering state-of-the-art is less than adequate* 
Here the concern is primarily for the training process and includes a num- 
ber of instructional assists for the student as well as the capability for 
student -controlled instruction (i.e* , self-initiated and to a degree, self- 
paced)* Underlying this approach is the utilization of the computer as an 
instructional instrument, specifically involving preprogramed training 
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TABLE 6. PILOT COMMAND ENTRIES BY TRAINEE 



Pushbutton 


Function 


Units 


Heading 


Ordered Aircraft 
Magnetic Heading 


0-360° Magnetic 
Resolution 1 


Airspeed 


Ordered Aircraft 
Indicated Airspeed 


0-400 Knots 
Resolution 1 knot 


Altitude 


Ordered Aircraft 
Indicated Altitude 


0-65,000 MSL 
Resolution 1 ft. 


Engine Start 


Ordered Aircraft 
Engine(s) Start 




Taxi 


Ordered Aircraft 
Taxi Sequence 




Commence* Flight 


Ordered Aircraft 
Flight Commencement 




Land 


Ordered Aircraft 
Landing Sequence 




Stop Engine 


Ordered Aircraft 
Engine(s) Shutdown 
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exercises i automated performance measurement, storage of performance 
information, provision for knowledge of results, and the control of infor- 
mation presentation to the student. 

A variety of design options, which are best described as deliberate 
departures from operational realism, are available. The intelligent use of 
these options (i.e. , integrating the training process with engineering fidelity) 
serves to enhance training effectiveness by providing conditions for learning 
to occur; these instructional assists are also a prime means of motivating 
students to perform. The information requirements relative to engineering 
design and the instructional merit provided by these capabilities are cata- 
logued below so that decisions about the relevance of an item to a proposed 
technical approach may be more systematically evaluated. 

2.3.4. 1 Display Enhancement. Use may be made of techniques which 
yield identifiable coding dimensions for enhancing the display of information 
to the student. Signal enhancement involves the modification of one or more 
of the stimulus dimensions defining the objects) of interest in a display. 
The concern is for augmenting the identifiable coding dimensions by which 
information can best be displayed to achieve the training objectives as a 
f function of the stage of instruction. Two types of instructional benefits 

>. accrue: 1) the probability is increased that the student will detect and 

classify events in cluttered displays as well as monitor signals effectively 
in complex displays (conspicuity and meaningfulness of signals); 2) guidance 
is provided to delimit the frequency and severity of incorrect responses in 
performance. Proper use dictates that display enhancement be under in- 
structor control so that it may be invoked during initial training with the 
response cues eventually withdrawn as training progresses until the student 
responds to the stimulus situation which describes terminal performance. 

The common techniques comprising this form of coding are the 

following: 

• color — a practical limit is seven colors; a limit of four is 
desirable to alleviate color registration problems 

• shape — shape coding is constrained by the ability to gener- 
ate characters 

• size- -four size levels are recommended as a maximum 
limit 

• intensity- -this is not desirable as a coding option except 
r as a means of attention-getting. Brightness contrast be- 
tween an object and its background has limited application 
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• orientation- -use of vectors, for example, is desirable 
for indicating heading, with vector length depicting speed 

. pulsing — pulsing signals may indicate action required 
and target traces may indicate previous position 

• alphanumeric --this offers unlimited possibilities and 
easily interpreted meanings can be assigned 

Selection of a specific technique must be made in the light of the display 
medium employed. CRT displays are most amenable to these techniques. 
Some form of temporal variation is preferred to steady state codes but for 
practical purposes they are equally effective in attention-getting power and 
in classifying multi-event situations. 

Display enhancement is appropriate as a means of stimulus 
support during initial training where the student is required to discriminate 
among information classes preparatory to action* It is particularly amen- 
able to computer -gene rated synthetic displays as a means of bringing the 
student's attention to an object or to an out-of-tolerance condition (e.g. , 
violation of minimum lateral separation of aircraft, critical weapon release 
distance) which demands an immediate and appropriate response. 

2.3.4.2 Performance Information Feedback. Techniques are available 
for providing information to the student about how the results of his perform- 
ance conform to expectations or norms. Of concern is the provision for 
augmented information feedback, via equipment, wherein the student is 
provided a signal during or immediately after responding which indicates 
the adequacy, correctness or accuracy of his performance. This information 
not in the learning task per se. This form of knowledge of results strength- 
ens and sustains performance and aids in eliminating established but in- 
appropriate responses. It is of particular value in indicating error to the 
student, for example, when error has exceeded criterion limits, when a 
procedural error has occurred, or when the trend of performance is ap- 
proaching an out-of-tolerance condition. The salutary effects of information 
feedback on learning and on performance, given immediately as accrued, 
are will known and the use of this capability in design is highly recommended. 

A number of design options is available for providing supple- 
mentary or augmented information feedback via training equipment. These 
involve computer control of the information feedback based on an automated 
monitoring, evaluation and scoring system which provides the signals for 
defining and selecting the message requirements as a function of student 
error in relation to preset tolerances. Selection may be made from among 
the following. 
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2.3.4.2.1 Error Indications on Primary Displays— This technique is 
applicable in situations where the student utilizes electronic displays in job 
performance. For example, in EW training, errors committed by the 
student in emitter signal analysis can be shown on the display as accrued 
Most suitable is the alpha-numeric message, providing only essential in-" 
formation so as not to clutter the primary display (e.g. ..an indication of 
equipment locus and type of error made). Variations on .this theme include 
the use of the following to indicate actual vs. desired values. 

. pointers— panel instruments may be modified to 
provide visual feedback. For example, a pointer - 
type instrument could have a redundant pointer 
added. This, feedback pointer would be smaller 
in width and length and of a different color to 
eliminate- ambiguity . 

. instrument indicator lights— this approach utilizes 
miniature lights built into an instrument face. When 
an error envelop is exceeded the lights are energized 
sequentially, showing a motion in the direction of the 
required action. 

. symbolic — feedback may be provided in the form of 
schematic representations of pertinent parameters 
such as found in Head-up displays (HUD). A varia- . 
tion of this for the HUD is to generate terse messages 
on a combining glass placed in the normal out- of- the - 
window field of view. 

. instrument drivers— this requires the activation of 
an instrument associated with the parameter requiring 
correction (e.g. , altimeter for altitude error; the 
HDI for pitch or roll error). A function can be devel- 
oped to drive the relevant indicator during feedback 
cycles, i.e. , driving the instrument from the actual 
value to the desired value on a cyclic basis. 

. peripheral indications— error readouts (indicator 
lights , digital readouts) may be positioned adjacent 
to the primary display in formats most meaningful 
to the remediation required (e.g. , the display of 
adaptive training scores). 

2. 3.4.2.2 Auditory Signals-It is desirable also to alert the student, 
via audio signals, to indicate, an out-of-tolerance condition at a given time. 
The design options include, alerting sounds and prerecorded messages. 
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• alerting sounds—this approach uses an audio frequency 
as the information feedback. The performance evalua- 
tion program, based on student response, selects an 
audio frequency band corresponding to the parameter 
out-of-tolerance. The deviation of the student's 
response from the desired response is proportional 

to the change in audio frequency from the center 
frequency of the selected frequency band. For example, 
in the flight simulator, when the pitch angle exceeds 
tolerances, the performance evaluation program de- 
termines that pitch angle is in error and selects the 
frequency band corresponding to pitch angle being in 
error. 

. prerecorded messages — several forms are viable. 
In one approach, the sequence of words/ syllables to 
compose the required message is stored digitally in 
computer core, disk, or drum. All possible messages 
are stored in memory, and the proficiency measure- 
ment programs will identify the required message, 
specifying the message number, code or address, 
the system will decode the number and thereby deter- 
mine the word sequence and output the correct sequence. 
Audio insetting is useful when only a small number of 
standard formats are required for the total message. 
The standard formats are stored on some continuous 
type of audio medium (tape, cartridge, etc.). The key . 
words describing the parameter of interest and the 
direction of deviation are inserted in the appropriate 
position. Continuous messages can be selected. In - 
this approach, each feedback message is stored as an 
addressable item, and for each feedback message de- 
sired, a storage slot is required. 

• computer- generated sounds based on voice synthesis — 
this approach requires the storage of phonetic sounds 
and a message is composed by the sequencing of the 
correct sounds. Each message stored contains the 
sequence of sounds and the spacing to construct 
messages. 

2.3.4.3 Guidance Supports. Cues and prompts provided by equipment 
(i.e. , information given prior to performing) axe particularly useful during 
initial learning by emphasizing the relevant aspects of performance, and 
by limiting the exploration time and minimizing gross errors in perform- . 
ance. Generally, this provision of subsidiary information is more appro- 
priate to part-task training devices than to complex weapon system trainers 
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which place a premium on engineering fidelity and on the direct transfer of 
training from the device to the operational system . Pertinent guidance 
techniques are indicated in the following, 

2.3.4.3. 1 Demonstration — This capability enables the device to exhibit 
proper or desired performance of selected system behavior patterns, man- 
euvers, or emergency procedures. An enactment of some portion of the 
training is provided to show a desired solution, an example of a relevant 
event, or the actual execution of a maneuver (idealized of "school solution 11 ). 
Such computer-controlled demonstrations (student or instructor initiated) 
provide examples of the necessary technical aspects to insure student under- 
standing prior to his actual performance. A prerecorded verbal explanation 
and commentary may accompany the demonstration to enhance the instruc- 
tional quality. In flight simulators, a demonstration can be executed such 
that cockpit instruments and the motion module exhibit movements as desired 
for the maneuver or flight segment selected. This may be in real time or 
slow time or combinations of both. 

Similarly, instructional advantage is obtained by the demon- . 
stration of electromagnetic signal specifications in various EW mission con- 
texts. The student gains an understanding of the job requirements from ex- 
amples of signal signatures across the range of emitter types under the 
variety of reception conditions in the EW environment. 

The helicopter instrument trainer, device 2B24 1 provides a 
number of demonstration flight segments which emphasizes instructional 
quality as well as technic accuracy. Examples of types of demonstration 
programs (provided in both real and slow time) are listed below. 

• interpretation of attitude and flight control 
instruments; navigation instruments. 

• various approaches including reporting procedures 
and execution of a missed approach. 

- VOR 

- tactical ADF 

- standard ADF 

- ILS 

• aircraft control during climbs, descents, turns 
and level flight by reference to attitude instruments. 

• holding at an intersection, including entry from 90°, 
off-axis and departure at a predetermined time. 

^TDC, Specification for Synthetic Flight Training System, Device 2B24. 
RFPN6 1339-69-0044, Proj. 1955, Orlando Florida, 1968. 
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• inflight engine failure and air restart* 

• instrument takeoff. 

Providing the demonstration capability requires a consider- 
ation of the following: 

• the aspects of the mission (maneuvers, segments, 
lesson plans) to demonstrate* 

• the number of, and the length of each demonstration. 

^ • the points in the mission cycle or lesson where 

> demonstration is available* 

1 • the means for switching from one demonstration 

I to another* 

1. the initiation of, and control of, the demonstration 

\ speed (real time, slow time) 

2.3*4*3.2 Predictor Instruments --This design technique involves an 
information display which presents future or predicted information based 
on manual contrbl performance of the moment* The feasibility of this 
approach has befen demonstrated in a number of research studies of track- 
ing skills (see Smode 1971). An example of this is a predictor display which 
provides an indication of the response characteristic of a system in terms 
of predicted system output at some time in the immediate future* Providing 
the student information relative to the future of the variable being controlled 
simplifies considerably the continuous control task -and thus provides instruc 
tional support during initial learning. The rationale for this option is the 
same as that indicated for display enhancement. 

2.3.4*4 Trainer-Peculiar Equipments. Trainer-peculiar or extra- 
mission equipments refer to hardware and associates training functions 
which are not present in the operator stations in the actual syst 
Their purpose is twofold: 1) to increase training flexibility by p_ viding 
the student the opportunity to select and initiate certain instructional 
alternatives as well as kllow self-paced instruction consistent with the 
mission requirements and the level of training; and 2) to enhance 
learning through the provision of supplemental performance information 
not available on the job. This facet of instructional control provides 
an assist to the student for optimally achieving the training objectives 
in a sequence of instruction. The utilization of these assists is most 
efficient when under instructor control in a planned strategy of instruction 
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for the individual student. Items of equipment which emphasize the "train- 
mg tool" characteristics of simulation are described below. 

2. 3.4.4. 1 Specific Stimulus Control. In complex training environments, 
stimuli associated with various sequences of performance are not easily 
identified. This hampers the control of the instructional process since 
the instructor is not able to correlate specific stimuli and behaviors. This 
problem is acute in training devices employing large instructor-to- student 
ratios, particularly when several stimulus inputs are combined in the 
performance. Means are available for presenting and monitoring specific 
stimuli crucial to defined performances. Specific preprogramed stimulus 
questions may be displayed at selected times to the student which re- 
quire a defined behavior or output. The response is monitored by the 
instructor who usually will have available several action alternatives 
relative to developing an instructional strategy for the student. In 
training devices employing automated monitoring and scoring, with 
programed error tolerances for defined performances, the instructor 
is also able to probe for causes of violations (excessive error) by 
selecting and presenting pertinent stimulus questions to which the 
student must respond. An example of this form of stimulus control 
is utilized in the Air Navigation Trainer, Device 1D23 (described in 
paragraph 2.3.3.2). An annunciator panel displays preprogramed 
questions, cues or probes, and a performance status indicator provides 
immediate knowledge of results of how the performance requested con- 
forms to expectations . 

The design for specific stimulus control is based on the 
following information requirements. 

. specific critical stimulus questions, , cues, probes 

. number of questions, cues, probes to be employed 

. length and format of each stimulus item 

. places in the preprogramed exercise the stimuli 
are presented 

. time length for the display of each message 

. design of the stimulus display (panel) 

. form of, and the display of the knowledge of results 

. keyboard provision and design for student response 

. record of the responses for exercise critique 
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2.3*4.4.2 Procedures Monitoring Displays. It is desirable to provide 
the student immediate knowledge about the adequacy of performing lengthy, 
sequential procedural operations which are a part of the total job require- 
ments. What is required is a display pf the checklist or procedural sequence 
to be* accomplished and some indication of the adequacy of the procedural 
performance as accrued (e.g. , in-tolerance performance and errors of 
omission and commission in checklist completion). This is particularly 
appropriate to Operational Flight Trainers wherein the student must per- 
form a number of precise checklist sequences involving both normal and 
emergency procedures throughout a mission cycle. In addition to the com- 
puter software requirements, a cockpit display is required (e.g., a CRT 
or a vertical screen) and a means for formatting, modifying and updating 
the procedures monitoring information. 

An example of such a capability is the Trainee Monitoring 
System (TMS) currently employed in OFTs. Although the TMS serves as 
a performance measurement technique and now is only provided at the 
Instructor Station, it may also be installed in the trainee cockpit to provide 
supplemental information about performance not available in the job context. 
The trainee Monitoring System and the information requirements for pro- 
cedures monitoring displays are described in detail in Chapter 3.4.8 in the 
section on Instructor Station Design. 

2.3.4.4.3 Student Control of Instruction. It is desirable to provide the 
student greater control of events in training exercises than has been the 
traditional case. This involves a capability for initiating or deleting certain 
instructional events in order to enable the student to tailor problem diffi- 
culty to perceived needs or ability of the moment (self-pacing capability) or 
to the level of training being accomplished (based on instructor decisions 
defining the options available at a given time in the training sequence). 

The design for student control of any instructional situation 
should provide the following capabilities (as required). 

• freeze and unfreeze the trainer 

• initiate and terminate preprogramed demonstrations 
in real time or slow time 

• select and initiate training exercises or segments of 
an exercise (as appropriate) 

• engage/ disengage the platform motion system 

. modify the initial conditions for a given training 
exercise 
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• alert the instructor that assistance is desired 

• reset the trainer to ths programed initial conditions 
for an exercise 

• select displays of performance as appropriate (e.g. , 
scoring information, error display) 

• moiify exercise difficulty by disabling an external 
forcing function (e.g. , turbulence), or disabling a 
vehicle characteristic in order to simplify the con- 
trol task (e.g. i select a constant altitude in order 
to concentrate on heading control) 

• place the device under automatic control wherein 
it performs within real-time programed tolerances 
or continues the steady- state condition prevailing 

at time of initiation (e.g. , automatic co-pilot control 
of a simulated aircraft when the student is .required 
to perform tasks incompatible with the control of 
the aircraft) 

^ 2.3.4.4.4 Student Information Display. Correlated with the problem 

control capability is the requirement to provide feedback of status and 
performance information. The following student information display capa- 
bilities should be provided in any design, as appropriate: 

• indications of student-initiated control of instruction 

- problem freeze 

- motion system off 

- instructor acknowledgement of call for assistance 

- device is in demonstration mode 
•p training exercise selection 

system status and performance information feedback 
indications 



c 
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performance level or score 

error out- of- tolerance 

adaptive training score (as appropriate) 

student may (now) assume control in training exercise 

automatic playback of performance is in progress 

student takeover from automatic playback condition 

device is under automatic control (performing either 

within real-time programed tolerances or continuing 

the steady state condition prevailing at time of initiation) 
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Display configuration 



- information formats and size 

- panel position in the trainee station 

- layout of the information array 

I'i^L 4 * , l° iCe AlertS * THe 8tud6nt may be alerted by means of audio 
signals that performance is outside of acceptable tolerance limits. Voice 

recorded "° Bt P*> minent * in OFT.. For example a 

recorded voxce wfll xndicate, via the intercom, when critical fliaht pkram 

:l e gm e ( nt ff 'mgnt 88iVe a " -ce for a given 

The design considerations include the following. 

. critical parameters per task or segment of the 
mission profile 

. error tolerance envelopes per maneuver or flight 7 
condition 

. means for determining message priority in out-of- ~\ 
tolerance performance U/ 

. message content and length 

2 3.4.4.6 Command and Response Capability. This refers to means for 
-XT* ^ C ° m P Uter * achieve of the instructional features de- 

to ttt m f thl « 8ectlon ' In e88e ^e, the concern is for hardware capability 
to transnut student actions or decisions, i.e. , data entry to or response to 
xriformatxon or commands fr.om the computer. The keyboard, emplZ 
pushbuttons in various arrays, is the technique currency most a^ropH^e. 
™Lol?r a T r 8 ?° nSe Ca P ability in trainee station served several 
are to b^'served ^ requir6ment8 ~* as a func tion of which- purposes 

r*™«»« ♦ «. a * Provi8ioni8 made f °r the student to recommend flight 
regime, to the computer; all flight changes are inputed through a keyboard. 
Tins purpose 18 served in the previously cited device 1D23 Air Navigation 
Traxner, wherein the student is provided controls and indications TorTe 
followxng: aircraft heading, altitude and speed control; engine(s) start and 
stop; and taxa, takeoff and landing. In general, the option affords the student 
he response capability for ordering commands although he is not a part of 
the dynamic response loop. For example,, an EW trainee conducting an 
evasxve maneuver to an airborne interceptor, simply orders (selects) a A 
maneuver V1 a keyboard input; the simulated own-aircraft completes the J 
maneuv-r and automatically returns to the previous track 
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b. Provision is made for the display of performance infor- 
mation to the student, e.g., the results of his response to interrogations 
from the computer. An example of this is the display of student output in 
response to programed messages (see item 2.3,3.3). 

c. Provision is made for direct data entry to the computer, 
as applicable, to achieve the performance requirements in an exercise. 
Included also are requests for information needed in exercise performance. 
The design for the command and response capability should account for the 
following information requirements as relevant to the needs of the device 
under consideration. 

• special keyboards vs. general multi-purpose 
keyboard 

• functions and commands required 

• values or units represented 

• characters and symbols represented 

• keyboard(s) array or matrix requirements 

• coded overlays required for modes of operation 
. keyboard(s) size and position in trainee station 

• keyboard(s) utilization procedures 

2.3.5 Post Note . The design of most complex training devices usually 
involves more than one level or approach to fidelity of simulation to achieve 
the desired instructional capability in the trainee station. 

There are subsystems requiring high fidelity in physical Simula- 
tion to provide the desired direct transfer of training from the device to the 
operational system counterpart. 

Deliberately backing off from duplicating the characteristics of the 
operational situation is also a prevalent design option to achieve cost econ- 
omies by reducing engineering fidelity without compromising training effec- 
tiveness. 

Where the engineering state-of-the-art is less than adequate, we 
are forced to accept reductions in tolerances that are the best achievable. 

There are also deliberate departures from realism which attempt 
to integrate training process and engineering fidelity. This differs from a 
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cost-effective lowering of the engineering fidelity or a reduction in fidelity 
tolerances where the engineering state-of-the-art is less than adequate and 
is concerned with incorporating training techifolcjy into design. 

Depending on the purpose of a training device and the basic design 
philosophy (replicas of operational counterparts, generalized trainers, part- 
task devices), no given level of fidelity is uniformly preferred at the ex- 
pense of other options, and advantage accrues from employing combinations 
of the approaches mentioned above. 

The four cases described in this section reflect the feature that 
several approaches may be incorporated effectively into a complex simula- 
tor. However, the four cases do not cover every situation since there are 
so many classes of training device. Also, human factors technical approach 
inputs in instances where the engineering state-of-the-art is less than ade- % 
quate, suffer in accomplishment and hence the fashion is to accept current 
design practice with the rejoinder that further research is required. The 
supporting research requirement appropriate to this situation is discussed 
in Section IV of this report. Basically, however, the material provided in 
this section lends support to fee human factors specialist in organizing the 
information requirements relevant to the engineering design of trainee ^\ 
stations . 1*J 
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SECTION III 



INSTRUCTOR STATION DESIGN 
3. INTRODUCTION 

Substantial human factors contributions are made in instructor station 
design. Here, the emphasis is on the application of training technology to 
design, vis-a-vis behavior modification and transfer of training. The major 
aspect of design concerns the management of training involving the display, 
control and communications requirements in setting up, controlling, moni- 
toring and evaluating instruction. The design requirement is to provide a 
capability for structuring training so that the definitive mission and system 
events can be installed and controls provided to insure that these events 
occur in prescribed ways at prescribed times. The bridge between student 
performance and instructional strategy is the measurement capability. 

This section presents concepts related to, and techniques and procedures 
for defining the information requirements pertinent to instructor station 
design. The resultant information array serves as a basis for examining 
engineering alternatives (hardware and software design options) for achiev- 
ing the instructional capabilities articulated in the MC . 

Two concepts underly the human factors guidelines presented in this 
section and they are tacitly assumed in the discussions which follow. The 
first is the desirability of automating a number of key instructor functions. 
This instructional assistance enables more precise control of the instruc- 
tional process and enhances the potential of a device as a training tool. The 
specific benefits subsumed under this concept are explicated throughout this 
section. The second is the desirability of the generalized (modular) con- 
struction of the instructor station. These two features, more than any other 
enhance the potential for flexibility and economy in design via the installation 
of a fundamental set of hardware (digital computer, I/O devices, etc.) and 
a software design pertinent not only to an individual device but to a class of 
training device. This enables substantial modifications in instructional 
capability within a given device or across similar devices without radical 
change in instructor station configuration. As the need for revision arises, 
the hardware is still applicable; only the software programing is modified, 
tailored to the new instructional objectives. Although the state-of-the-art 
has not advanced sufficiently to suggest these as ordinary design options, * 
they nevertheless represent the ' oal to be achieved as data and technique 
are forthcoming from research and application. 

3. 1 MANUAL ADAPTIVE TRAINING CAPABILITY, As a prelude to the 
discussions which follow, it is worthy to note that the manual (normal) adap- 
tive training capability is a desired human factors design goal for complex 
training devices (within the immediately foreseeable state-of-the-art); it 
most closely satisfies the assumptions and instructional goals set forth in 
this report. 
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The manual adaptive capability is * f fined by the options available to 
the insti *tor for keeping each student p >gressing through a sequence of 
instruction at that student's own achiever -at speed. The value of this 
(particularly in developing strategies to enhance the instruction of a group 
of students undergoing simultaneous but independent training) is the flexi- 
bility provided in the control of problem difficulty level and in the amount of 
instructor intercession required in effectively shaping individual behaviors. 
.This approach takes advantage of computer assists in instruction, but relies 
on the instructor to initiate and control the actions required in developing 
instructional strategies tailored to the individual student. This is in con- 
tradistinction to computer controlled automation of the instructor functions 
wherein problem difficulty is automatically adjusted as a consequence of the 
immediately preceding performance of the student (see Chapter 3.4.10). 

To achieve this capability, the following automated assists are provided. 

m Standardization of Training - -Preprogramed le=son plans are 
provided involving a series of standardized exercises graduated 
in difficulty with a capability for instructor intervention in pre- 
scribed ways to control the training process. This includes the 
override of defined mission events to accommodate students who 
• perform below expectations for the given exercise and for the 
insertion of additional events in an exercise to accommodate 
students who exceed scenario requirements. 



Computer Assisted Measurement System — Automated evaluation 
and scoring with scoring criteria adjusted to the stage of training 
of each student provides error indications and information which > 
are displayed at the instructor console. The instructor is able | 
to select information on specific students on demand. 

Automated Monitor and Control Capability- -Compute r - driven 
multi-format QRT displays are provided at the instructor con- 
sole presenting performance/error and mission status information 
in all relevant modes and in variable formats (both alpha-numeric 
and graphic). 

Instructional Controls — Controls are provided at the instructor 
station for monitoring student performance and system status 
and for initiating, continuing or modifying mission event'* to 
achieve desired training strategies . 

Hard Copy Records of Student Performance — Printouts of student 
pertormance information (time and event data, error deviations 
which exceed programed envelopes, summary information) are 
available on demand for w-s during and after a training exercise 
(for critique and for school record-keeping purpose 
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The provision of these automated assists enables considerable flexi- 
bility in the shaping of desired behaviors. In addition to tailoring the learn- 
ing progression to each student f s abilities, variations in stimulus support 
can be provided as a function of stage of learning. For example, in early 
learning, maximum assistance can be provided all students as required 
(e.g., problem freeze, guidance, demonstration, reinstruction) overlaid 
on the standard mission scenario. As student proficiency increases, the 
stimulus supports are gradually withdrawn until the standard scenario pre- 
vails. Problem difficulty exceeding the requirements of a given scenario 
may also be installed to accommodate students who perform above the scen- 
ario standards* 

Thus, the instructor is provided the opportunity to continually monitor 
all critical aspects of student performance. He is primarily a monitor and 
evaluator of student performance and has the control capability to alter 
features of the mission scenario commensurate with the training strategy. 
The basis for this is the automated performance monitoring, evaluation, 
and scoring system. The automated assists enable the following instruc- 
tional options: 

• Continue preprogramed mission, no action required. 

• Override of defined programed mission events. 

• Provide feedback of performance information to selected 
student(s) based on monitored information obtained from 
the displays. This may be in the form of knowledge of 
results of performance or cues to the student for enhancing 
performance. 

• Halt the problem for any or all students and provide guidance, 
or halt the problem until all students have completed a mission 
phase; resume mission. 

• Demonstration mode- -provide a capability to demonstrate 
aspects of a task or maneuver to all students or to demonstrate 
specific characteristics of performance for any or all students 
during briefing o when performance is below expectations 

for the specific mission number; resume mission* 

• Reinstruction mode — provide a capability to return to an 
earlier portion of the mission for any or all students when 
performance is below expectations for the mission. The 
reinstruction squires that the student perform again that 
which he ha? just completed (i.e. , a segment, leg or portion 
of a leg of f i>o mission). 
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Error alert mode — when a student exceeds the error envelope in 
the mission for a class of error, a display of this information 
is provided the instructor and a computer-initiated problem freeze 
is a conventional outcome* The instructor options include: 
manual override of the freeze; accept freeze with procedures 
to continue (including the setting of new error tolerance envel- 
opes); as well as those listed above in order to develop the 
training strategy for the student in question. 

Insert new events into the exercise --when a student's perform- 
ance exceeds the mission scenario requirements, the instructor 
has the option of inserting new events in addition to those in the 
preprogramed scenario. This provides for a controlled increase 
- in problem difficulty for the purpose of keeping any student at the 
threshold of his ability at any given time. 
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3.2 BASIC APPROACH. Consistent with Section II, the human factors 
effort is to determine the information requirements w hich engineering 
must account for in proposing technical approaches to the design for in- 
struction. The derivation of the information requirements and the instruc- 
tional capabilities relative to instructor station design is based on the range 
of functions performed by the instructor. Basically, the analyses concern 
the following: what information is required for monitoring the total training 
situation: what controls are needed to display appropriate information to 
the student and to select appropriate information about student performance; 
what instructional options are available to the instructor in monitoring, 
evaluating and controlling training; and what modes, formatting require- 
ments, speed and flexibility of displays are required to enable instructional 
control? 

The display, controLand communications requirements are also ex- 
amined in light of the functions performed in pre-exercise setup, enroute 
exercise operations and post-exercise operations. Once completed, these 
information requirements are utilized by the human factors specialist in 
collaborating with engineering specialists to determine the technical ap- 
proaches for implementation consistent with the nature of the contemplated 
training system. 

The remainder of this section is devoted to an analysis of instructor 
functions and the information requirements subsumed under each function 
as a basis for determining the design of the instructor station. Table 7 
presents a listing of the range of instructor functions that must be accom- 
plished (pertinent to most devices in the NTDC inventory). The lieting is 
generic; any given device does not invoke the total list of functions. Each 
function is, however, pertinent in some measure to one or more devices 
or device classes. For each of the functions listed in Table 7, a descrip- 
tion is given of where in the training exercise or total training sequence it 
pertains, whether the function is specific to a training device class. or gen- 
eral to most training devices, and what instructional capability or level of 
instructor involvement is provided by the function. 

The information requirements associated with each of these functions 
are discussed in the detail necessary to permit the correlation of these 
needs with proposed engineering solutions. The reader should determine 
which of the functions (and the specific instructional implications) pertains 
to his specific design situation, concentrate on these and ignore the re- 
mainder (this is in keeping with our intent to provide "guideline" support 
for most any training device situation). 
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3.3 INSTRUCTOR FUNCTIONS FLOW ANALYSIS. When the specific 
instructional functions to be provided are identified t Table. 7) and prior to 
beginning the detailed analyses of the information requirements per function, 
the human factors specialist should develop an overview of the instructor 
involvement in the training system in order to consolidate thinking about 
the design philosophy and to aid in the further definition of design require- 
ments for the instructor station. This is accomplished in the form of a 
functions flow analysis which depicts a summary of the desired sequence 
of instructional operations and the options available to the instructor in 
developing a training strategy". This enables the human factors specialist 
to "put together" the candidate design features for instructor involvement 
in structuring and controlling training. Figure 9 provides an example of a 
functions flow analysis. The example is for an airborne Electronic Warfare 
Training System. This is semi-automatic system (4 to 1 student-to- 
instructor ratio) having a number of computer assists in instructional capa- 
bility. The device characteristics include the following: 

. preprogramed mission scenarios graduated in problem 
difficulty 

• automated monitoring, evaluation and scoring capability 

. computer driven CRT displays for presenting all relevant 
performance and mission information; 

r off-line modification and generation of scenarios 

. hard-copy printouts of student performance 

. manual adaptive capability (the instructor has the means for 
adjusting task difficulty so that it is appropriate to each 
student's level of skill; the preprogramed scenarios are 
modifiable enroute enabling the achievement of a strategy 
for training both for the above and the below standards 
performer). 

The functions flow diagramjs continually improved throughout the 
instructor station information analysis phase. It serves importantly as a 
check for design in that 1) the information requirements are correlated with 
the operations in the diagram, and 2) all the blocks depicted in the final dia- 
gram must be accounted for in -the selected engineering approaches. Thus, 
the functions flow diagram serves as a human factors "test and tryout" 
vehicle prior to interactions with the engineering specialists. It also serves 
as the initial point of contact between human factors and engineering in 
defining the technical approaches to device design, i.e. , "role play" the 
scenario with engineers to place the total instructional system in perspec- 
tive prior to more detailed considerations. 
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3.4 INFORMATION REQUIREMENTS BASED ON FUNCTIONS PER- 
FORMED* The specific display, control and communications requirements 
that influence the engineering design choices for a contemplated training 
device are detailed ne*t. - These are presented as guidelines for the human 
factors cialyses of the information requirements that must be accounted 
for in the decision to initiate the development of a training device. In pro- 
viding inputs to a specific technical approach study,, the human factors 
specialist examines only those functions described in this section that per- 
tain to the device under consideration (as specified in the MC documentation). 

The information analyses requirements are presented in a series of 
chapters; each chapter considers a basic commonality in content which is 
related to specific functions performed in the instructional process. With- 
in each chapter, relevant design options are identified and analyzed. The 
analysis centers en: the use and training value of the design option, the in- 
formation requirements that must be accounted for in implementing design, 
and a discussion of the assumptions and the constraints involved in engineer- 
ing implementation. 

The congeries of design issues are portrayed in the remainder of this 
section. Fourteen separate chapters are devoted to articulating the infor- 
mation requirements pertinent to the structure and control of training. The 
chapters are organized as follows: _ 

# Off-line Preparation for Trainer Use 

* » Training Problem Formulation and Presentation 

# Data Handling and Display 

. Information Display Requirements for Monitor, Evaluation 
and Scoring of Performance 

# Control Capabilities for Monitor, Evaluation and Scoring of 
Performance 



o 



% Instructional Options Capability 

# Measurement System Design 

# Procedural Performance- -Display and Recording 

# Capability and Controls for Inputs to the Computer 

. Software Capability for Developing Automated Adaptive 
Strategies 



■ERIC. 
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• Automated Verbal Messages 

• Communications 

. Post-Exercise Instructional Capability 

• Human Engineering Design 

In addition, a summary of the information requirements relevant to 
training device design is provided in a checklist format in Table 8. This 
sets forth tiie sequence of activities to be accomplished in order to achieve 
■ fac * 0r8 .| n P ut8 in 8u PP° rt oi the Technical Approach determination. 

Each information item in the series is individually considered via the appro- 
pnate column in the checklist. In addition, the specific paragraphs in the 
text which are appropriate to each item in the checklist are identified. This 
correlation of checklist items with text material provides a convenient 
reference for the human factors specialist. The checklist serves as an aid 
for organizing the information requirements prior to interacting with en- 
gineering specialists in technical approach development. 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN 



Checklist Item 



Accounted 



for 
(Y) Yes 
(N) No 




Pertinent 
Paragraphs 
in the Text 



A. Off-line Preparation for 
Trainer Use 

1. checkout of the trainer 

2. training problem selection 
and setup 

3. modification of existing 
programed scenarios 

4. generation of preprogramed 
scenarios 

'5. readiness for on-line 
operations 

B. Training problem Development 

1. job content definition 
(preprogramed exercises) 

2. organization of training 

- exercises (preprogramed) 

3. definition of initial conditions 

4. means for installing initial 
conditions 

5. automated briefing 
(pre-exercise) 

C. Trainer Operation 

1. off-line trainer operating 
modes 



3.4.1.1, 3.4.9 
3.4. 1.2 

3.4. 1.2, 3.4.9 



3.4.1.3, 3.4.2, 
3.4.9 

3.4.1.4 



3.4. 2. 1.1 



3.4. 2. 1.2, 
3.4.2. 1.3 

3. 4. 2. 2 

3.4.2,2 



3.4. 2.3 



3.4. 1. 1, 
3.4.7.1.1, 
3.4.7.2, 3.4.8.1 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



Checklist Item 


Accounted 
for 
(Y) Yes ! 
(N) No 


N/A 


| Pertinent 

Paragraphs 
1 in fFiA Tpvf 

[ ill tile X wAl 


2. 


on-line modes of 
operation 








| 3.4.2, 3.4.3, 
| 3.4.5.1, 3.4.O. 

1 4 ft ^ 4 Q 

3. 4. 12, 3. 4. 13 


D. Automated Assists for 
Instructional Control 










1. 


preprogramed standard 
lesson plans 


\ 


■ 




3.4.2 


' 2. 


cozmputer assisted 
measurement system 








1 A n 


3. 


automated monitor and 
control capability- 








*l A *1 1 A A 

3.4.5 


4. 


automatic error alerts 








3.4. 6.4. 1 


5. 


autopilot program 








3.4.6.4.3 


6. 


hard copy performance 
records 








.3.4.7, 
3.4.13.1 


E. Instructional Control Options 










1. 


selection of display system 
modes 








3.4.2.4, 3.4.5, 
3.4.6.1 


2. 


selection of displays for moni- 
toring and evaluating student 
performance 








3.4.5. 1 


3. 


modify preprogramed scenarios 
divising and exercise 




* 




3.4.2, 3.4.3 
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TABLE 8. CHECKLIST OF/ INFORMATION REQUIREMENTS 
FOR INSTRUCTOR.STATION DESIGN (Continued) 



Checklist Item 



Accounted 

for 
, (Y) Yes 
! (N) No 



N/A 



Pertinent 
Paragraphs 
in the Text 



4. guidance supports' 

«. demonstration programs 

• reinstruction 

5. specific stimulus presentation 

6. assists in problem control 
. automatic error alerts 

• individual parameter freeze 
. autopilot program 

7. supplemental assists 
(e. g. , CCTV) 

8. knowledge of results 

9. hard copy records of 
student performance 

Specific Display Requirements 

1. trainee station displays 

2. device status and operation 

3. auxiliary information 

4. communications 

5. monitoring and recording 

6. mission displays 

7. visual factors in displays 

• legibility requirements 
. speed and flexibility in 

adjusting display content 



3. 4. 6. 2 
3.4. 6. 2. 1 
3. 4. 6. 2. 2 

3. 4. 6. 3 

2.4.6.4 
3. 4. 6. 4. 1 
3. 4. 6. 4. 2 
3. 4. 6. 4. 3 

3. 4. 6. 4. 4 



3.4. 6. 4.5 
3. 4. 13.1 



3. 4. 4 

3.4.4 

3.4.4 

3. 4. 12 

3.4.7 

3.4.3.3 

3.4. 3. 2. 1, 
3.4.3.2.2 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



Checklist Item 



8. procedural (checklist) 
• performance displays 

* G » Information Content of Displays 

I 1. mission data 

i 

2. display of student history 

3. mission/scenario area 

4. own-vehicle and support/ 
friendly vehicles 

5. mission events /installations 
- 6. target data 

7. total situation display 

8. environment 

9. console operating modes 

10. indication of system state or 
mode of operation and phase * or 
point in the mission or exercise 

11. location (geographic) of all 
students in the exercise 

12. variance of vehicles about 
the ideal track 

13. status of events initiated 
in the trainee station 



Accounted 
for 
(Y) Yes 
(N) No IN/A 



Pertinent 
Paragraphs 
in the Text 



3.4.8 



3.4.3, 3.4.4 
3.4.3, 3.4.4 
3,4.3, 3.4.4 
3.4.3, 3.4.4 

3.4.3, 3.4.4 
3.4.3, 3.4.4 
3.4.3, 3.4.4 
3.4.3, 3.4.4 

3.4.2, 3.4.3, 3.4.4 

3.4.3, 3.4.4 

3.4.3, 3.4.4 
3.4.3, 3.4.4, 3.4.7 
3.4.3, 3.4.4 



ERIC 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



Checklist Item 



Accounted 
for 
<Y) Yes 
(N) No 



N/A 



Pertineut 
Paragraphs 
in the Text 



14. indication of student/t^eam 
control or tactical action and 
consequences of the action 

15. indication (and time) of 
student/team detection 
and identification of target, 
from onset 

16. indication of instructor actions, 
initiated and under control 

17. instructor control of vehicles 

18. performance information on 
each student or team (error/ 
status /activity) 

19* indication of out-of -tolerance 
performance 

20. indication of vehicle(s) out- 
of -tolerance ' 

21. indicatic of adequacy of student 
procedural performance 

22. indication of student/team 
error deviations from pro- 
gramed tolerances 

23* types, number of communi- 
cations errors 

24. indications of actions re- 
quested of a student 



3.4.3, 3.4.4 



3.4.3, 3.4.4 



3.4.3, 3.4.4 

3.4.3, 3.4.5 
3.4.3, 3.4.4 

3.4.3, 3.4.7 
3.4.3, 3.4.4 



3.4.3, 3.4.4, 
3. 4.8 

3.4.3, 3.4.4, 
3. 4.7 



3.4.3, 3.4.4, 
3.4. 12 

3.4.3, 3.4.4 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



3 



Checklist Item 



25.. indication of malfunctions/ 
emergencies/failures 

26. indications of critical time 
and event happenings during 
exercise 

27. st«uus information requests 
from the student 

H. Control Functions 

1. trainer and display system 
* mode selection (o&'-line 

and on-line) 

2. scenario generation or 
modification (off-line) 

3. initial condition setup 

4. ancillary and power supply 
controls 

5. mission/instructional control 
(enroute exercise) ' 

6. control of procedural (checklist 
performance 

7. inputs to the computer 

L Specific Control Capability for 
Monitoring, Evaluating and 
Scoring Performance 

1. device operation 



Accounted 
for 

(Y) £ es 
(N) No 



N/A 



Pertinent 
Paragraphs 
in the Text 



3.4.3, 3.4.4 



3.4.3, 3.4.4 



3.4.3, 3.4.4 



3.4.3.4 

3.4.1.2, 3.4. 1,3, 
3.4.9 

3. 4. 2. 2 

3.4.5 

3.4.5 

3. 4.8 

3.4.9 



3.4.4, 3.4.5 



3 



ERJC 
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TABUS 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



Checklist Item 



Accounted ; 
for f 
(Y) Yes . 
(N) No JN/A 



Pertinent 
Paragraphs 
in the Text 



2, trainee station controls 

3, means for displaying 
student history 

4, means for positioning units 
in an exercise 

5, environment (media) controls 

6, means for inserting and 
manipulating targets' in an 
exercise 

7, means for inserting parameters 
and events at prescribed times 
in an exercise 

8, means for inserting non- 
programed events or elim- 
inating programed events in 
preprogramed scenarios 

9* means for selecting CRT 
display modes and formats; 
page selection 

10. " selection of specific perform- 

ance information (e. g. , indiv- 
idual/all students, error data, 
system performance data) 

11. means for freezing individual 
parameters in an exercise 

12. means for inserting /denying 
failures or emergencies 



3.4.5 

3.4.3., 3.4.5 

3.4.5 

3.4.5 
3.4.5 

3.4.5, 3.4.6 
3.4^5, 3.4.6 



3.4.4, 3.4.5 



3.4.3, 3.4.5 



3.4.5, 3.4.6.4.2 
3.4.5 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



"1 

Accounted 
; for 
1 (Y) Yes 

Checklist Item i (N) n q 


N/A 


Pertinent 
Paragraphs 
in the Text 


13. means for direct access to 
device computation system " 


t 

f 

z 
t 




3.4.5, 3.4.9 


14. communications co^rols 


i 

j 
i 




3.4.12.3, 3.4.5 


15. means for controlling target 
or support vehicles 


i 
1 

i 

! 


- 


3.4.5 


16. controls to tally defined com- 
plex responses of students 


I 

► 

i 

\ 




3.4.5, 3.4.7 


- 17 # means for recycling the sys- 
tem in automated exercises 

• reinstruction 

• remedial branching 

• recvcle students who pyppaH 
scenario requirements 






3.4.5, 3.4.6 

■ 


18. means for initiating and 
terminating demonstrations 






3.4.5, 3.4.6 


19. means for selecting perform- 
ance information to be scored 
and recorded 






3.4.5, 3.4.7.5 


20. means for inserting changes in 
present error tolerances 






3.4.5, 3.4.7 


21. means for monitoring and con- 
trolling student procedural 
performance 






3.4.5,3.4.8 


22. means for selecting hard copy 
records of student performance 






3.4.5, 3.4.7.5. 2, 
3.4. 13. 1 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



i 

Checklist Item 


i ■ 

Accounted 
for 
(Y) Yes 
(N) No 


N/A 


Pertinent 
Paragraphs 
in the Text 


23* interlock controls between 
instructor station and TDO 
console (to avert unauthor- 
ized insertion of additional * 
parameters into preprogramed 
scenario) 


■ 




3.4.5 


24. means for changing, or modify- 
ing mission areas 






3.4.5 


J. Measurement System Design 








1. selection of what to measure 






3. 4. 7. 1 


2. selection of measures 






3.4.7.2 


.3. establishment of performance 
standards 






3. 4. 7. 3 


4. establishment of performance 
tolerance envelopes for com-* 
puter scoring 






3. 4. 7. 3 


5. capability for modifying com- 
puter scoring criteria 






3.4.7.3.1 


6. sampling of performance 
strategies ' 






3.4.7.4 


7. performance data sampling 
rates 






3.4.7.4. 1 


8. organization of the measure- 
ment data 






3.4. 7.4.2 


9. display of measurement 

information during the exercise 






3.4. 7.5 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 

FOR INSTRUCTOR STATION DESIGN (Continued) " 



Checklist Item 


Accounted 
for 
(Y) Yes 
(N) No 


! 

N/A 


Pertinent 
Paragraphs 
in the Text 


10. hard copy computer outputs 






3.4.7.5.2, 3.4. 13. 1 


11. automatic quality control of 
training data 






3. 4.7.6. 2 


J\. jji splay ana Control of Proce- 
dural (Checklist) Performance 








1. modes of operation 






3.4.8. 1, 3. 4. 8. 2 


2. definition of the procedures 
and the step sequences for 
each procedure 


- * 




3.4.8. 2. 1 


3. display presentation formats 






3.4.8.1.3, 
3.4.8. 2. 1 


4. control requirements 






3.4.8. 1.3, 3.4.8. 2 


1j. Data Entry to the Computer 








1. off-line modes of operation 






3.4.9 


2. enroute problem control 






3 "4 9 


3. display modes associated 
with the input device 






3.4.9 


4. definition of sequences in 
operating the input device in 
all modes 






3.4.9 


M. Automated Adaptive Simulation 
Capability 








1. selection of adaptive variables 






3.4, 10 


2. selection of measures 






3. 4. 10. 4 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



Checklist Item 



I 

Accounted 
for 

T) Yes 
N) No 



N/A 



Pertinent 
Paragraphs 
in the Text 



3. establishment of error 
tolerances 

4. selection pf an adaptive logic 

5. establishment of the perform- 
amce sampling periods 

6. means of displaying perform- 
ance information 

N. Automated Verbal Messages 

1. selection of appropriate 
instructional situations 

, 2. definition of task structure 

3. selection of appropriate 
word universe 

4. definition of number of mes- 
sages required and message 
structure 

5. assignment of message priorities 

6. selection- of timing sequence for 
successive messages for the 
same system state 

7. means for modifying messages 
and for overriding programed 
messages 



3.4. 10.4 

3.4. 10.5 
3.4. 10.5 

3.4. 10.6 



3.4. 11 

•3.4.11 
3.4. 11 

3.4. 11 

3.4. 11 
3.4. 11 

3.4. 11 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



A 

* 

Checklist Item 


I 

» V» vvUUbvv 

for 
lY) Yes 
N) No 


NT/A 


Pertinent 

Pa ra.cr Tr» r\K c 

in the Text 


O. Communications 








1. extent of representation 






3.4. 12. 1 


2. instructor monitoring capability 






3. 4. 12. 1 


3. display of communications statue 






3.4. 12. 1 


4. capability for common audio net- 
work, simultaneous communica- 
tions or extra vehicle' communi- 
cations 


i 




3.4. 12. 1 


5. operating controls 


• 


- 


3.4. 12.3 


6, controls for recording audio 
performance of the student 






3. 4. 12. 3 


7. controls for activating pre- 
recorded verbal messages 






3.4. 12.3 


P. Post -Exercise Instructional 
Capability 








1. types of hard copy perform- 
ance records required 

• critique 

* quality control 






3.4. 13. 1 


2. information to be recorded 






3. 4. 13. 1 


3. formatting of retrievable 
records 






3.4. 13. i 


4. means for selecting programed 
performance variables, sam- 
pling rateB and formats 






3.4. 13. 1 
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TABLE 8. CHECKLIST OF INFORMATION REQUIREMENTS 
FOR INSTRUCTOR STATION DESIGN (Continued) 



1 

Accountec 




1 




for 




Pertinent 


* 

Checklist Item 


^ i / x es 




Paragraphs 


(N) No 


N/A 


in the Text 


5. video recording and nla.vKa.rk 






3.4. 13.2. 


capability 








• selection of content 








i means -of control for start 








and exit in playback mode 








• selection of playback mode 








speed ' f 




- 




• means to access desired 








portions of the video 








record for playback 








6. pictorial reconstruction of just 






3.4. 13.3 


completed exercises (fcr critique) 








7. audio records for critique 






3.4.13.4. 


Q. Human Engineering Design 








1. application of human engineer- 






3.4. 14, 


ing criteria and design recom- 






mendations as specified in 








MIL STD 1472A, MIL H 46855 








and other appropriate specifi- 








cations 








• layout and configuration 








• display and control groupings 








, and positions 








* panel markings 








• lighting 








• workspace constraints 

■ 
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3.4.1 OFF-LINE PREPARATION FOR TRAINER USE 

A number of pre-mission operations must be considered in deter- 
mining the technical approaches to design. Of concern are the functions 
performed by the instructor (and/or the training device operator) off-line, 
prior .to initiating a given training exercise. The design issues involve 
the information requirements for the following: 

* checkout of the trainer 

» problem selection and setup 

• generation of new scenarios 

. readiness for on-line operations. 

Figure 9 provides an example of the kinds of pre-mission instruc- 
tor operations performed for a semi -automated EW trainer prior to mission 
initiation. It also indicates the options available for recycling the trainer 
to another previous stage or to another exercise in the training sequence 
instead of shutting the trainer down. Figure 10 provides another example 
of the possible operations to be performed by the instructor (or TDO) prior 
to mission initiation for device 1D23, Air Navigation Trainer. 

3.4.1.1 Checkout of the Trainer. The instructor is required to monitor 
and control the trainer checkout to determine visually that the simulator is 
ready for operation. This is accomplished in accordance with the Daily 
Readiness (DRED) Check programs. Each of the trainer systems to be 
tested are activated by simulated inputs, with system outputs compared 
with predetermined values. The information requirements for design 
include: 

, Selection of automatic or manual operation in the 
DRED mode. 

• Layout of the function keyboard overlay for the DRED 
mode. Figure 11 depicts a DRED mode overlay proposed 
for device 2F101, T-2C Operational Flight Trainer 
(Goodyear 1971). 

• The procedures and associated controls for individual 
system tests, total system test and fault isolation. 

\ 

• The content and format of the test results displayed on 
the CRT and on hard copy printout (as required). 

3.4.1.2 Problem Selection and Setup. This off-line function includes 
the selection and mounting of an existing preprogramed scenario or the 
modification of an existing scenario to achieve a current training objective. 



-106- 




WfiBLB 
FOR Ptomod 




5BLEC7 
flPD/T/OATfiL 
PflRfiM£7E*5 





Modify 




Problem 




PARMeTBAS 




Kor mo 



No 



Modify 

KOR BNO 



1 






REQUEST 




COMptTTER 




STORAGE. 




OP 




Measures 



,t\Jo 



MOO/FY 
RftOlo 



5T0D£NTS 
TO 

INSTRUCTORS 



NAVTRADEVCBN 71-C-0013-1 



>- 





{ 














» 




Mode. 






Posmoi* 



> 



ASSIGN 

ST ODE NTS 
To 

INSTRUCTORS 
AND TDO$ 



A'o* 




Modify 






Modify 








Wbasvres 







- MOOlFf 



SELECT 

^Tftc Mode 
jsus Mopes 

CflOTS OR 



WfrHTiW* 
COCKPITS 





ERJC 



Figure 10. Instructor Premission Operations Envisaged for 
Device 1D23 Air Navigation Trainer. 
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Figure 11. - DRED Mode Overlay 

(from Goodyear 1971) 



For existing scenarios, the design requirements include: means 
for mounting the preprogramed scenario for on-line use; a capability for 
reviewing scenario presentation in summary; and controls for initializing 
the trainer for mission operation. 

To modify existing scenarios for current use, means must be 
provided to alter lesson plan parameters or modify the processing flow. 
An example of this caoability is provided in the specifications for device 
2F101 (T-2C OFT) (NTDC 1971). The PLAN mode of operation enables the 
instructor to change the programed order of checklist procedures , alter 
programed mission legs or segments and delete or add to the surface faci- 
lities. A proposed PLAN mode overlay on the function keyboard used in 
conjunction with a light pen and CRT is shown in Figure 12 (Goodyear 1971). 
For example, the instructor is able to change a mission leg by rewriting 
the maneuver to be performed, the parameters to be monitored and the 
ending of the leg* To achieve this, the following steps are accomplished 
with the PLAN mode overlay. 

« Depress PROGRAMED MISSION switch on the function 
keyboard. 
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. Display the legs to be changed by typing on the keyboard 
. the programed mission number and the leg number for 
all the legs to be changed. 

. Change the leg parameters by typing on the alphanumeric 
keyboard the programed mission number, the leg number, 
the maneuver to be performed, the mission leg end, and 
the parameters to be monitored. 

* . Depress ENTER switch on the function keyboard. The 
display of the legs to be changed and the typed alpha- 
numeric information will be displayed on the CRT in the 
appropriate format. 
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1 


pointer 
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FACILITY 
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PROCEDURE 
MODIFIED 
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DELETE 
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PROCEDURE 

DISPLAY 

RIGHT 

4 
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Figure 12 - PLAN Mode Overlay 
(from Goodyear 1971) 
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An example of an off-line editing capability for modifying an 
existing set of preprogramed problem parameters is provided by device 
2B24. It represents a navigational aids editing sequence. The complete 
typewriter routine for making the desired navaids change is shown below. 

NAV AIDS EDIT 

STATION TYPE (UHF, FM, VHF, VOR, ADF, ILS, GCA)? VOR 
WANT LIST OF CURRENT STATIONS? N 
WANT TO EDIT? Y 
WHAT LINE NUMBER? 2 
2 SPARE 

WANT CHANGE? Y 

REPLACE/ ADD/ DELETE (R, A, D)? A 
INPUT DHN 
DONE? Y 

WANT STATION PARAMETERS? Y 
WHAT STATION NUMBER? 2 

VOR CALL DHN 

LNOl FREQ 
LN02 XPOS 
LN03 YPOS 
LN04 RNGE 
LN05. ALT 
LN06 MV 

WANT TO EDIT? Y 
WHAT LINE NUMBER? 1 

1 FREQ 

WANT CHANGE? Y 

WHAT VALUE? 111.6 
DONE? N 

WHAT LINE NUMBER? 2 

2 XPOS 

WANT CHANGE? 16.85 
DONE? N 

WHAT LINE NUMBER? 3 

3 YPOS 

WANT CHANGE? -36.35 
DONE? N 
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WHAT LINE NUMBER? 4 

4 RNGE 

WANT CHANGE? 120 
DONE? N 

WHAT LINE NUMBER? 5 

5 ALT 

WANT CHANGE? 400 
DONE? N 

WHAT LINE NUMBER? 6 

6 MV 

WANT CHANGE? 2.2E 
DONE' ? 

WANT NEW STATION NUMBER? N 
WANT NEW STATION TYPE? N 

JOB DONE 

EDIT COMPLETE-READY TO STORE DATA 

3.4.1.3 Generation of New Scenarios . An off-line capability for instruc- 
tor generation of new scenarios is required when appropriate existing pre- 
programed scenarios are unavailable. The information requirements • 
pertinent to the design of software and hardware to accept, interpret, store 
and execute the instructional inputs include the following. 

. Establish the exercise type and level of task difficulty 
(where the exercise belongs in the total training se- 
quence) and identify the tasks, maneuvers and segments 
to be taught. 

. Establish scenario structure ("problem world, facilities, 
vehicles, targets, media) 

. Establish the parameters and their quantities/ values 
(e.g., environmental conditions, flight paths, vehicle 
parameters, mission events, etc.) 

• Identify the malfunctions to be incorporated, the in- 
sertion points and the probabilities of malfunction 
insertion in automatic modes 

. Define the performance measures and enable the display 
and recording of all performance and error information 
available in the computer 



-111- 



NAVTRADEVCEN 71-C-0013-1 



• Specify the error criteria and tolerance envelopes for 
\ performance, and modifications in error criteria as 

a function of student performance 

• Provide performance demonstrations and specify the 
replay rules and conditions 

• Establish' the guidance and feedback supports and the 
rules for student assistance (prompts, cues, knowledge 
of results) 

. Provide automated aural briefings (as appropriate) 

3.4. 1.4 Readiness for On-Line Operations. Design must also make 
provisions for additional operations which occur just prior to initiation of 
an actual training exercise. This type of instructor control capability is 
exemplified in the following. 

• In multi-student trainers involving a number of instructors 
provide means for the assignment of selected students to 
given instructors 

• In multi-student trainers, provide means for the selection 
(and indication) of trainee stations in "ready" status 

• In general trainers, provide means for selecting the 
pertinent active mission displays and information inputs 
in the trainee station (e.g. , displays and controls perti- 
nent to a reconnaissance mission vs. a defensive mission 
in EW training). 



-112- 



KAVtRAdeVCEN 7i-C~0013rl 



3.4.2 TRAINING PROBLEM FORMULATION AND PRESENTATION. 

th* • Mea T ns " lu8t be Provided for the installation of training content in 
the device. Instructor involvement centers about two basic design options- 
automated preprogramed lesson plans, or the manual setup of inftial con 
ditions for an exercise (including the capability for preproLamed^l 
cordon data sets involving instructor selection of a gfven daU set" The 

Z ^c^nient% COn ? rn8 - e8t f bli8hing ***** *" bl ™ Parameters anl 
the content of instruction for each exercise in a sequence of training. 

3.4 2.1 Preprogramed Training Exercises. The human factors -inputs 
include the specification of the content of the instructional sequence, the 
number of, and variations in the exercises and the grading of the content 
set on a continuum of problem difficulty ranging from basic indoctrination 
training objectives to advanced complex performance sequences. The 
stringency of the performance standards for the exercises is usually cor- 
related with the difficulty continuum. 

The information requirements that must be fulfilled to achieve 
a lesson plan library involve the following. 

*. 4. 2. 1. 1 Job Content. The range of specific tasks and performance 
conditions that define the content of training must be defined. This con- 
siders the total task, mission segments or maneuvers that are to be 
installed for training. Two examples are provided here to indicate the 
information requirements pertinent to design. The first concerns the job 
content for the Air Navigation Trainer, Device 1D23 (NTDC 1971). The 
preprogramed lesson plans essentially call for a one-to-one correlation 
with the job of the navigator in the air environment. The exercises are 
constructed to enable the student to apply directly what he has learned 
and experienced in the device to realistic air navigation problems. The 
content centers on typical naval air operations involving basic air navi- 
gation, navigational systems operation, fuel management, and radio 
communications. The content range is identified below. 

• Basic dead- reckoning navigation— This involves 
the utilization of VOR, ADF, TACAN, VORTAC, 
and doppler and inertial navigation aids. Emphasis 
is placed on basic enroute navigation procedures, 
navigational flights, calculations of wind effects, 
and fuel management (and includes relevant mal- 
functions in navigational equipments). 

• Airways navigation— This involves low altitude 
(victor) and-high- altitude (jet) airways navigation. 
Job requirements include: general procedures 
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C 



c 



1 



c 



in airways navigation; airways interceptions; 
position establishment via navaids (ADF, VOR 
TACAN); communications (IFR, VFR reports); 
fuel management (including. bingo time, fuel 
and distance), ETA and weather problems; and 
critical enroute problems (e. g. , airspace 
violations resulting from drops in barometric 
pressure, jet stream variations, etc.). 

• Operational navigation missions* * Typical 
operational missions are installed including: 
a bombing sortie involving segments of cata- ■ 
pult launch, proceed to rendezvous point, 
engage target, return to carrier and carrier 
controlled approach; and a standard bombing 
sortie but with a scenario wherein the carrier 
i* is gone when the aircraft returns, requiring 
a search for the carrier (search time depen- 
dent on fuel remaining) and proceeding to the 
divert field and landing. Training objectives 
include: position determinations (via doppler/ 
inertial systems), fuel management, ETA, 
communications, and various system mal- 
functions inserted as required. 

The second example illustrative of the information require- 
ments for preprogramed training exercises is. provided by the Synthetic 
Flight Training System, Device 2B24 (NT DC 1968). A series of exercises 
for training in instrument flight is provided; each exercise is made up of 
fifty contiguous, problem segments. The student is required to perform 
the range of instrument flight maneuvers during the total training program. 
Each problem segment consists of a steady state condition and the transi- 
tion from it to another steady state condition. It begins with a set of initial 
conditions and sufficient additional data to describe ideal performance 
during the exercise on such performance parameters as airspeed, altitude, 
flight path, rater of climb, rate of turn, bank and pitch altitude, and on 
engine parameters. Malfunctions (e.g., in electrical system, engine 
system, flight control system) are included in the programed scenarios. 

3.4.2. 1.2 Organization of Exercises. Once the content is determined, 
the exercises are organized individually and across the sequence of training 
to provide a coherent program of instruction. In devices which provide a 
direct transfer of training to the operational counterpart, individual exer- 
cises are organized according to the relevaut mission profiles or scenarios. 
In generalized or part task: devices, exercises are organized in terms of 
the functions defined in the training objectives. 



ERIC 
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The number of exercises required to achieve the desired 
library is specified. This includes the range of different exercises 
desired as well as the variations on each theme. 

3. 4. 2. 1. 3 Problem Difficulty. A key feature in training content 
development is the organization of exercises in a manner most advanta- 
geous to the shaping of student behaviors. The.most useful scheme is to 
provide a defined series of exercises graduated on a continuum of diffi- 

n^I'e of aChi6Ve ^ in 6ne ° r * combination of ways, depending on the 

nature of the device and on the instructional requirements. 

• E ^sy to ha rd continuum — This organization is based 
on the nature of the task and on a continual increase 
in the events provided in training (and, therefore, 
on the demands placed on the student), Involved is 

a logical progression of training content from simple 
to increasingly more difficult sequences of perform- 
ance. For example, in Device 1D23, the progression 
of instruction begins with indoctrination exercises 
and continues with basic dead- reckoning navigation, 
and ends with navigation exercises in representative 
operational mission contexts. 

• Procedura l to fully integrated continuum— This diffi- 
culty dimension is reflected in an exercise series 
wherein initial exercises are concerned with proce- 
dural adequacy. From this, more complex training 
objectives are installed (in which previously accom- 
plished objectives are included in that they repre- 
sent building blocks for more complex activities). 
Thus, later exercises emphasize full utilization of 
the portions of the system trained in earlier exer- 
cises. 

• Conditions of performance- -Task difficulty may 
also be manipulated by providing increasingly 
stringent conditions on performance. The classes 
involved here include: the speeds of event hap- 
penings (e.g. , faster own-ship and target speeds), 
environment degradation (e. g. , variable winds 
and speeds, variability in water conditions), and 
stimulus degradation (e. g. , intermittent targets, 
garbled communications). 
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• Error tolerances — Task difficulty may be increased, 
all other things equal, by imposing increasingly 
stringent error requirements on adequate perform- 
ance, as a function of stage of traiaing(e*g. , pre- 
cision demanded in manual control responses), 

• Stimulus supports— Similar task sets may be varied 
in difficulty as a function of the amount of stimulus 
supports built into an exercise (e. g. , prompts, cues, 
knowledge of results, stimulus enhancement). 

• In some situations a graduated difficulty continuum 
is not an issue. The requirement is to provide ex- 
posure to a range of events and happenings which 
occur in the operational counterpart. This recog- 
nizes that the ordering of materials is as important 
as the difficulty progression in learning. Certain 
job^situations involve the acquisition of a set of skills 
before additional skills can be mastered and these 
may be no more difficult than the preceding set. 
This is particularly so in cognitive tasks and in 
verbal learning. The familiar part-task to whole- 
task learning progression is meaningful here* 
Thus, this approach favors situations where subtasks 
are progressively added to the learning situation so 
that the amount of information learned increases as 

a prelude to handling other aspects of the job. The 
human factors requirement is to 3pecify the range 
of these events so that they may be simulated in the 
appropriate instructional contexts (i. e. , coverage 
of all relevant events). 

3. 4. 2. 1. 4 Performance Standards. The standard of adequate perform- 
ance and the error tolerances for each exercise must be stipulated (this 
is discussed in paragraph 3.4.7. 3 in the chapter on Measurement System 
Design). 

3.4. 2. 1. 5 Scenario Modification. Means must be provided for 1) off- 
line generation or modification of scenarios, and 2) on-line intercession 
into the 4 preprogramed scenario to provide the flexibility to develop an 
instructional strategy tailored to, or at least pertinent to each student. 

3. 4. 2. 2 Manual Installation of Initial Conditions. Insertion of problem . 
parameters to the simulation program is inade at fhe instructor console 
when preprogramed exercises are not warranted. This may be accomplished 
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via individual controls for setting up the desired initial conditions, or via 
the selection of a preprogramed initial conditions data set. An example 
of each is described next. 

The ASROC/ASW Early Attack Weapon System Trainer, Device 
14A2 (NT DC 1964) provides for the individual parameter setup of the initial 
conditions for each exercise. The classes of controllable parameters that 
are installed at the instructor console are the following: 

• Own- ship 
course 
speed 

position (XY) 
equipment malfunction 

• Support units 
range 
bearing 
course 
speed 

altitude (aircraft) 

• Target 

position (including contact at prescribed time) 

range 

bearing 

course 

speed 

depth 

type submarine (via speed) 

aspect 

doppler 

false target 

• Environment 
sea state 
layer depth 
wind 

• Weapons 
ASROC 
DASH 
AWTT 

nuclear depth charge 
aircraft torpedoes 
submarine torpedoes 



-117- 



NAVT fcA&fiVCEN 7 1 -C-00 13- 1 



These manipulables are employed, as required, in a series of training 
exercises graduated in difficulty. Three groupings (blocks) of training 
exercises are utilized. The first block centers on basic attack proce- 
dures (non-mission context) in which the objectives specify the develop- 
ment of fire control solutions and launch procedures for the relevant 
weapons. The next block of objectives is devoted to single ship sequences 
in the screen mission context where the previous block requirements are 
imbedded in a total mission from search and detection through weapon 
assignment and target prosecution. The final block deals with multi-unit 
missions (Search and Attack Unit (SAU) context) involving more complex 
tasks of coordinating the operations of other units. This represents the 
fullest expression of coordinated team training objectives. 

The A-7E Operational Flight Trainer, Device 2F100, provides 
for instructor selection of preprogramed sets of initial conditions. At 
least ten of these preselected data sets are stored in computer memory 
and are addressable at the instructor console. In addition, the capability 
is provided to permit the instructor to change the values of these data sets. 
Verification of the existing content is also provided via a hard copy print- 
out of initial condition data. 

The preprogramed initial conditions data sets include the 

following: 

• aircraft position coordinates 

• altitude 
f velocity 

• heading 

• attitude 

• fuel 

• oxygen 

• weather conditions and aircraft environment 
(wind, barometric pressure, rough air, icing, 
magnetic variation) 

• radio aids changes program (select station 
number, type, parameter and data) 
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• chart changes program (select appropriate 
chart and parameters) 

• recorder modes selection (mode of operation, 
approach station and approach parameters) 

• takeoff and landing simulation program 
(catapult takeoff, normal takeoff, arreted 
landing, normal landing) 

• also, preprogramed values are inserted for 
computer control of parameter deviation (for 
automatic performance recording systems), 

3.4. 1. 3 Pre-exercise Briefing. The automated briefing of students, 
while in position in the trainee station, is a desirable option'to accompany 
preprogramed training and evaluation (e.g., checkride) mission scenarios* 
An automated library of briefing materials and control signals for demon- 
strations (e. g. , tape cartridges) properly loaded and indexed to correct 
start points is the design goal. The selectiou of the briefing events depends 
on the purpose of the training exercise and on the level of learning the 
student is at (exercise number in the total sequence of learning). 

The information requirements for achieving this capability 
include the following: 

a) The purpose and scope of the training (or evaluation) 
exercise, including a description of the mission profile 
and the procedures involved (problem content). 

b} Articulation of the training objectives 

• performance elements— e. g. , catapult launch; 
search and attack options; heavyweight takeoff; 
weapon actions, communications-navigation 
requirements such as required in instrument 
flight operations; etc. 

• conditions of performance --environmental 
conditions; emergencies and malfunctions; 
partial panel (flight) operations; aircraft con- 
figuration. 

• measurement— emphasis on what will be scored 
or what is required in performance; indication 
of automated adaptive scoring (if applicable). 
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• performance criteria— indication of what is 

expected and the error tolerances for successful 
performance in the specified exercise. 

) Guidance and instructional supports provided 

. demonstration capability provided (based on the 
description of the lesson and what i 8 expected 
of the student). 

. student capability for direct request for assist- 
ance, e.g., request for position (lat/long) 
information; redaction in task difficulty (e.g. , 
freezing a control parameter, such as holding 
altitude constant inflight). 

. capability for reinstruction on scenario aspects 
performed below expected tolerance limits 

. capability for student control of instruction 
fstudent-paced instruction) —the instructional 
options provided the student are selected from 
the following: 

- fre 

eze/unfreeze the trainer 

- initiate/terminate preprogramed demonstra- 
tions in real/slow time 

- select or initiate training exercises or segments 
of an exercise 

- platform motion system on/off 

- select displays of performance as appropriate 
(e. g. , error display) 

- modify exercise difficulty by disabling an 
external fcrciig function (e.g., turbulence) or 
disabling a vehicle characteristic in order to 
simplify the control task (e. g. , select a con- 
stant altitude in order to concentrate on heading 
control) 6 

- reset the trainer to the programed initial con- 
ditions for the exercise 
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d) Support material provided— the materials pertinent to 
the given exercise are made available to the student 
(e.g., route maps, radio facilities and ground stations 
and their frequencies, doctrinal publications such as 
ATP-1, etcj. 

e) Communications capability between student and instruc- 
tor . 
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3.4.3 DATA HANDLING AND DISPLAY 

The instructor station display system must provide complete train 
ing data on all students during an exercise and enable the monitor and con- 
trol of the pertinent aspects of performance. This is a crucial aspect of 
design, for the effectiveness of the instructional system is predicated, on 
this capability. The complexities in providing effective data handling and 
display requires considerable human factors effort in determining the in- 
formation requirements for design. 

The human factors inputs to the technical approach determination 
are influenced by a series of criteria which must be tacitly assumed in de- 
riving the information requirements for design. These criteria concern: 

• Display flexibility to call up any relevant aspect of 
student performance or problem feature on demand, 
with no detrimental lags in information presentation . 

- mission/ geographic areas 

- scale variations 

- information classes 

- display modes 

- coding of displayed information. 

• Adequate update of performance information. 

• Uniformity of information display in characters/symbols 
in multi-format contexts* 

• Commonality of data locations in multi-format contexts. 

• Information. handling capability (amount, types, rapidity 
of change, number of students under control). 

• Multi-format requirements, high density information 
display, information integration (alpha-numeric and 
situational/ graphic display requirements involving 
discrete event and overall situation displays; continu- 
ous alpha- numeric error readouts). 

• Rapid mode changes for monitoring, control and evalua- 
tion, requirements for monitor and control of more than 
one* student sequentially. 

9 Accuracy, reliability and registration (information 
alignment) of the visual displays. 
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• Expansion capability (for future display requirements), 

• Visual factors 

- legibility 

/ - refresh rate 

- contrast sensitivity 

- resolution 

- clutter 

The cathode ray tube display (with appropriate accessing controls) 
is the most efficient means for satisfying the design criteria* The CRT 
system utilizes the computer to present and update the classes of informa- 
tion required at given times in the exercise. This type of display is- capable 
of preaenting all of the critical training information, as required, in a small 
viewing area. In short, the CRT is able to provide more understandable 
arid instructionally meaningful information in the shortest time period than 
any other display. means. It is for these reasons that the discussion on the 
information requirements for data handling and display centers on the CRT 
system. No material is provided in this chapter on other types of informa- 
tion displays which are also involved in instructor station operations, 

The information requirements must consider a variety of design 
features, notably the display modes and locations, visual factors, informa- 
tion formatting requirements and the precision of the displayed information* 
Accordingly, the discussion considers the following categories: 

• display content and formatting requirements 

• visual factors in design 

• specific display requirements 

• controls for display selection and modification 

3.4.3.1 Display Content and Formatting Requirements* The display in- 
formation must provide sufficient content and organization to enable the capa- 
bility for monitoring and evaluating all critical aspects of the mission cycle 
or lesson plan and for modifying the training environment to shape student 
behaviors commensurate with demonstrated performance. Unfortunately, 
the information requirements for design cannot be specified in a straight- 
forward manner because of the diversity in content requirements not only 
across classes of training devices but also in terms of the specifics rela- 
tive to given trainers within a class. 

Accordingly, to provide useful guidelines for design, the ap- 
proach elected is to .restrict this chapter to provide: 1) only information 
that has some commonality to a number of display situations, and 2) display 
examples representative of a range of situations of interest to the human 
factors specialist. 
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All pertinent aspects of student performance and system status 
must be available for display either continuously or on demand. This in- 
volves the display of performance and status information in all operating 
modes for the specific trainer, both on-line and off-line. Of concern are 
the various modes of trainer operation for monitoring and evaluating student 
performance during an exercise (see Paragraph 3.4. 6,1) and the display 
modes required for off-line operations. The off-line operations, only iden- 
tified here, include the following modes: 

• daily readiness mode — capability to monitor and control the 
daily checkout of the trainer (see Chapter 3.4. 1). 

• problem setup mode— capability for generating preprogramed 
lesson plans or modifying existing ones, and selecting lesson 
plans for the training exercise (see Chapter 3.4*1). 

• inspect and change mode- -capability to inspect and change 
the library of instructor-controlled parameters in the pre- 
programed lesson plans (see Chapter 3.4.1). 

• replay mode --capability to replay the training mission in 
real-time for performance critique and in fast-time for 
search and freeze, 

3.4.3. 1.1 Display Requirements. During the exercise (enroute) a 
variety of displayed information is required to support the instructional 
decisions and actions. The following determinations must be made to achieve 
the display content and flexibility in data presentation: 

• display format for each mode— This involves the group- 
ing and positioning of the displayed information with an 
emphasis on commonality in format and data positioning 
across the modes. 

• type of display- -Both graphic and alphanumeric informa- 
tion and combinations of both are required. 

• events to portray per display—This refers to the range 
of events pertinent to the device under consideration and 
includes such groupings as error classes, students who 
exceed preset tolerances, flight track and performance 
information on all students, etc. 

• CRT page formats* -This includes the presentation of 
sequential data in successive pages or independent pages 
of data. 
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. the data common to each display mode must be identified; 
uniformity of information display categories in multi- 
format situations is required to avoid ambiguity and 
clutter and to minimize search time. 

. size of the mission area— This refers to geographic 
scale and the capability to select desired scales. 

0 

. continuous display of information vs. on-demand 
presentations. 

. specific information— This refers to the display of 
information specific to the environment represented 
in the trainer (e.g. , classes of EW emitters radiating, 
aircraft under control, etc.), and to stylized information 
for depicting defined relations (s.g. , bargraphs, per- 
formance data displays, etc.). 

The display of student performance and system status in- 
formation needed to maintain instructional control and to develop training 
strategies is as varied as the training devices currently on-line or in 
development. However, the various CRT display requirements may be 
organized in terms of the information needed in monitoring, evaluating and 
controlling performance and in the instructional functions performed during 
an exercise. These features must be clearly defined prior to deriving the 
display contents and formats and to determining what information is con- 
tinuously displayed, or selected on demand. Determining the content and 
organization of the displayed information must be anchored to the following 
instructional capabilities (as appropriate). Within these capabilities, guide- 
lines for design are provided through the use of selected CRT display exam- 
ples which indicate the range of the content displayed and the alphanumeric 
and pictorial formatting arrangements. 



• display of initial conditions at problem start- 
Figure 13 shows an alphanumeric format of a 
typical initial conditions display for a general- 
ized submarine advanced casualty ship control 
trainer (Lamb, Bertsche and Carey 1970). 

• demonstration mode. 

• monitoring and control of events throughout the ex- 
ercise—Figure 14 provides a display of flight track 
information on all students for an air navigation 
trainer (Bark, et al, 1969). A combined graphic 
and alphanumeric format is proposed. An exam- 
ple of a stylized display for monitoring frequency 
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Figure 13. Initial Conditions Display for Submarine 
Aivanced Casualty Slip Control Trainer, 
(from Lamb,.Bertsche and Carey 1970) 



coverage in an Airborne Electronic Warfare trainer 
is shown in Figure 15. In the monitoring mode the 
instructor may interrogate the computer via lightpen 
to determine the performance and mission status 
of each student irrespective of specific errors made. 

• reinstruction mode (see Paragraph 3.4.6.2.2). 

# event insertion--displays are required to provide 
information to assist in determinations of whether 
to insert additional events or delete programed 
events in a preprogramed scenario, based on 
student performance. Figure 16 depicts the air- 
borne intercept mode for the Electronic Warfare 
trainer cited above. Two approaches may be 
employed for displaying preprogramed airborne 
interceptor aircraft (A/ 1). In the first instance, 
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a programed A/ 1 will come on unless overridden; this 
is accomplished by the placement of the lightpen on 
DO NOT INSERT and EXECUTE. In the second 
instance, to insert a non-programed A/I into the 
problem, the lightpen is placed on INSERT and 
EXECUTE. The CRT display is an example of 
a pictorial format. 

error alert mode (see Paragraphs 3.4.6.4. 1 and 
3.4.7.5.1). 

performance assessment- -A variety- of formats are 
required to obtain performance and error production 
information. Figure 17 provides a display of per- 
formance data for a submarine advanced casualty * 
ship control trainer. This display graphically 
shows the response of the ship to various casualty 
and recovery actions. It may also be used fpr 
debriefing the student on the outcomes of perform- 
ance. Two forms of error production information are 
depicted. Figure 18 provides a continuous presentation 
of critical student er.rors, accrued throughout a 
mission. The example is from the above -cited 
Airborne Electronic Warfare trainer. Nineteen 
error classes are identified for display. The error 
groupings common to the basic reconnaissance and 
defensive missions in the mission profile are: 

- switch setup positions 

- equipment preset (bands/frequencies requirements) 

- switch setup for signal analysis 

- threat priority 

- evasive maneuvers 

- communications 

- switch shutdown positions 

The error groupings pertinent to the basic defensive 
mission are: 

- threat assessment 

- switch setup for jamming 

- initiation for jamming 

- use of expendables 

- environment coverage 
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Figure 17. Performance Data Display, 
.(from Lamb, Bextsche and Carey. 1970) 



The error groupings pertinent to the basic recon- 
naissance mission are: 

- signal setup for DF bearing 

- recording 

- analysis 

- subsequent DF bearing 

- priority search 

- general area reconnaissance 

- malfunction detection/ correction 

The alphanumeric display always shows the six most re 
cent classes of error made by a student. Each error 
class is allotted 2 lines on the CRT. Additionally, 
throughout the mission, a bargraph denotes the level 
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of proficiency per student (Excellent, Satisfactory, 
Unsatisfactory) and an error alert indication (when 
applicable) which notifies the instructor that the 
student has almost achieved the limit of error (that 
is i error limit minus one error), and that a com- 
puter freeze of the mission will occur when another 
error of the same class is made. Also, information 
designating the student number , mission number, 
and leg number is displayed continually. The 
second example is shown in Figure 19 which, pre- 
sents a page format for error analysis per student 
for the setup of switches in the same Airborne 
Electronic Warfare trainer. 

• malfunction mode-- Figure 20 presents a display of 
casualties appropriate to the submarine advanced 
casualty ship control trainer. The instructor is 
able to initiate individual casualties from this 
array and the pertinent actions and outcomes are 
displayed on the CRT. 

3.4.3.2 Visual Factors in Design. The specification of the CRT display 
requirements must also consider the visual factors which influence design. 
A sizeable number of visual issues are involved in equipment design (e.g., 
legibility, system resolution, brightness and contrast requirements, am- 
bient illumination, flicker, etc.). These elements are interdependent, 
resulting in significant interaction effects, for example, the minimum 
visual angle for character legibility increases as contrast decreases. An 
indication of some of the possible parametric interactions among the visual 
factors is shown in Table 9 (Semple, et al 1971). While not complete, the 
table serves to highlight the complexities involved in display system design. 

The information requirements for instructor station CRT display 
systems must consider a number of performance characteristics. These 
characteristics with recommended guidelines are outlined below. (The 
values provided represent current NTDC specifications.) 



0 



Performance Characteristic 



Minimum display refresh rate 



Minimum display update rate 



Recommended Value 



30 frames/ second (not interrupted 
by computer updating of display 
data) 

2 times/second 

(not affected by command inputs) 
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Performance Characteristic 


Recommended Value 


Minimum screen highlight luminance 


20 footlamberts 


jviinimum small cnaracter neignt 


0.15 inches 


Minimum large character height 


1,5 times small character height 


Minimum horizontal raster lines 
per character height if raster scan 
is used 


9 lines 


Character width to height ratio 


1:1.5 


Minimum small characters per 
horizontal line including spacing 


80 


% _T * . . * ____ __ _• 1 A. 

Minimum lines of characters per 
display page 


60 


Minimum number of intensity 
control levels excluding off 


2 


Display type 


Rectangular type vertically aligned 
CRT with rectangular display 
border mask 


Minimum usable horizontal display 


12 inches 


Minimum usable vertical display size 


16 inches 


Minimum horizontal address 
resolution 


1/750 (linear spacing) 


Minimum vertical address 
resolution 


1/1000 (linear spacing) 


Accuracy 


Within 2% of screen horizontal and 
vertical dimensions 


Drift 


Within 0*5% of screen horizontal 
and vertical dimensions over a 
period of eight hours with a +50° 
fahrenheit temperature variation 
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Performance Characteristic 


Recommended Value 


Spot size 


Less than 0,02 inches over the en- 
tire visible display area including 
the effect of jitter at a screen high- 
light luminance of 20 footlamberts 


Character repertoire 


At least 26 upper case alphabet, 
10 numeric, and 26 other special 
characters 


Lines and circles 


Dot, dash, dot-dash, and solid line, 
all hardware- implemented 


Flashing rate 


Once/ sec on I 


Flashing on/off ratio 


1:1 


Maximum time from display request 
to display of requested data 


3 seconds 


Polarity switch 


If raster scan is used, a polarity 
switch will provide a positive {dark 
characters on a light background) 
or a negative (light characters on a 
dark background) display. 



For our purpose, the information requirements for CRT display 
design center on those nerformance characteristics which influence legi- 
bility and the speed and flexibility in adjusting display content. The rele- 
vant display elements are described next* 

3.4. 3.2. 1 Legibility Requirements. A number of visual factors are 
subsumed under CRT display legibility. Those of importance in instructor 
station design are outlined below. Within each of the factors identified, 
some basic guidelines are provided for establishing the information require- 
ments for engineering design. 

• Viewing distance--The convention is to accept the 
maximum viewing distance (seated man) as 28 inches 
for operations normal to the line -of- sight. The 
minimum viewing distance from the eye to the dis- 
play should not be less than 1-1/2 times the width 
of the display surface. 
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STATUS 



CASUALTY 



SELECT 





1 FLOOD 


0 
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PRERECORD CAS NO* 12 




0 RESET 0 OPERATE 0 HOLD 0 INT CAS 0 CLEAR 



Figure 20. Casualty Display for Submarine 
Advanced Casualty Ship Control 
Trainer* 

(from Lamb i Bertsche and Carey 1970) 



♦ Viewing angle--The optimum viewing angle is 90°; 
it should not be less than 45° an d preferably at least 
60° in the horizontal plane. The location of displays 
above the horizontal line-of-sight should not exceed 
45°, however i no more than 15° is desired since 
head movement is restricted in this plane and fatigue 
is likely with continuous viewing . 

# Visual angle— A practical standard is for the stroke 
width of display characters to subtend a visual angle 
of not less than 2 minutes . Evidence indicates that 
characters are most legible when the ratio of char- 
acter height to stroke width is between 5:1 and 7:1. 

A simulator manufacturer (Singer-General Precision 
1969) has determined character size by the formula: 

Viewing Visual 
Character distance X angle 

size _ (inches) (minutes) 

(inches) ~ 3438.0 
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For example, with a viewing distance of 28 inches 
and a required visual angle of 15 minutes, a display 
character should be 0. 12 inches high as measured 
on the display surface. 

• Contrast — A method of measuring contrast ratio for 
CRT displays expresses contrast as follows: 

Bp + B B 

c R=-Bb— 

where 

B Q is the brightness of the written line 
when ambient light is excluded 

Bb i» the brightness of the screen from 
ambient light 

a contrast ratio of 2. 5 to 3. 5 to 1 is satisfactory for dis 
play systems operating in normal room ambience. A 
level of 3 to 1 is acceptable for single color displays. 

Refresh rate— The CRT display must be refreshed at a 
high enough frequency to avoid flicker. Thus, the re- 
fresh rate should be above the threshold for flicker per- 
ception (the lowest refresh rate at which no flicker is 
observed is the critical flicker fusion poirit— CFF). 
The critical frequency which satisfies the wide-angle 
vision at the instructor station is about 5 cps higher 
than that for foveal vision. Accordingly, a refresh 
rate greate* than 47 cps is required for a flicker-free 
display, for a short persistence phosphor (Singer- 
General Precision, 1969). Flicker may also be elimina- 
ted by reducing display brightness or using longer per- 
sistence phosphors. 

Jump phenomenon— When a short persistence phosphor 
is used, small quick eye movements may result in the 
observer momentarily seeing two images on the screen. 
This position jump is due to the stroboscopic effect of 
the pulsing phosphor on eye movements (about 20 milli- 
second duration) made gazing at a display. Jump is 6b-~ 
served on bright phosphor tubes with a character repeti- 
tion frequency of 30 to 60 cps , a character on-time of 
the order of 10-50 microseconds and a character decay 
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of 50% brightness in less than 100 microseconds 
(Singer-General Precision 1969). The phenomenon 
can be eliminated by an increase in scope persis- 
tence (e.g. , about 10 milliseconds for 50% decay) 
or by a decrease in the brightness contrast. 

• Form and spacing of characters— Character spacing, 
line spacing and character aspect ratio determine 
the packing density of characters on the display. 
Singer-General Precision (1969) suggests that the 
average width of characters should be between 65% 
and 80% of character height. Maximum spacing 
between characters should be 1/4 to 1/2 character- 
width laterally and 1/2 to 1 character-height ver- 
tically. The spacing between groups of letters and 
numbers should equal the average character width. 
Words written in lower-case characters tend to be 
less legible than those written in upper case; also, 
observer response time is faster in identifyirg upper 
case words. 

3.4.3.2.2 Flexibility and Speed in Adjusting Display Content. Consid- 
eration must be given to insuring that the information in the visual display 
can be arranged and presented in any manner suitable to instructional 
purposes. Since. all information cannot be displayed simultaneously, the 
instructor must be provided the flexibility to locate and call up the classes 
of information required for managing the shaping of behaviors without 
detrimental time lags in the changing of modes, pages or classes of per- 
formance or error information per student. Computer-generated, refresh 
CRT displays are most desirable for enhancing this capability (i.e. , not 
only in speed and flexibility requirements but also in providing good visual 
quality, accuracy, resolution and registration). 

3.4.3.3 Specific Display Requirements. The generic, instructor station 
is comprised of various classes of displays and associated controls. Sim- 
ply, these include: trainee station displays; displays providing device status 
and control (operation); auxiliary information displays, communications 
displays, monitoring and recording displays (e.g., CCTV, scoring readouts, 
hard copy) and mission displays. Our concern in this paragraph is for the 
information requirements pertinent to the design of mission displays. This 
type of display usually is specific, sometimes unique, to a class of training 
device. The displays are graphic /pictorial representations of specific 
portions of a training mission. Historically, mission displays have received 
greatest usage in flight simulation (e.g., graphic position plotters employ- 
ing pen recording, the more recent versions interfacing with the computer 
and receiving X, Y position data in the form of dc analog voltages and pen-up 
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and pen-down commands as discrete signals from the computer I/O). 
Current design utilizes the CRT to provide mission displays for showing 
vehicle ground track and position for various areas and scales . 

the info™ J^T CXampleS ° f mission Splays are provided next to illustrate 

f T T Te **™™<** pertinent to design. The first is the graphic 
plotting display system employed in the helicopter synthetic flight Sn B 

~)'nr C o^ e 2B24 (NT ^° 1968> - A *"P^*T (one for each SS? 

net conter a " *" gr ° Und track of *• associated 

helicopter against any of several preprogramed background maps and two 

sSLt.r.?. h / CpiCt time - based ^ts of altitudelnd airspeed? air 
folWng formation. The information requirements include the 

a. Ground track plotting-This involves the selection of 
appropriate scales and background maps for the cross- 
country mode (100 x 100 nautical mile area) and for the 
approach mode (10 x 10 nautical mile area). Cross- 
country mode characteristics include the following: 

• numbered airways with intersections 

• VOR stations and symbology 

• LF NDB stations and symbology 

• Marker beacons and symbology 

• ILS facilities 

• ground track vector of aircraft and ideal track 
. out-of-tolerance conditions shown by symbols 

at the point of occurrence on the ground track. 
The out-of-tolerance condition is marked for 
the following parameters— pitch, roll, vertical 
velocity, rate of turn, gas producer tachometer, 
rotor RPM, exhaust gas temperature, torque 
pressure, minimum fuel, icing airspeed exceed- 
ing VNE, altitude, airspeed, and ground track. 

Approach mode characteristics include: 

• runways (scaled length vectors) 

• obstructions and symbology 

• published clearance information 



b 



Airspeed and altitude plotting -continuous time-based plots 
of a.rspeed and altitude are provided. Each plot is of 20 
rmuutes duration, representing a point approximately every 
4 seconds which is the instantaneous value of the parameter 
Out-of-tolerance symbols are plotted along the time base to 
facilitate the correlation of the time plots with the ground 
track plot. 6 
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c. Status information-- The following information is provided: 

• Aircraft call number 

• Training mode (automatic, checkride or semi-automatic) 

• Student's name 

• Training problem in progress and difficulty level 

• Adaptive training score 

• Student history — read from punched card 

COMP - list of completed lessons 

TIME - average time to complete 

BA - below average time to complete 

AA - above average time to complete 

SCORE - composite error rate from completed lessons 

• Freeze mode indication 

• Partial Panel indication that one or more cockpit instru- 
ments have programed malfunctions active 

• Simulated copilot indication - when the simulated copilot 
is active 

• Assist-student indication - flashes when the student 
presses his INST CALL button. The indication is 
removed when the instructor acknowledges the request 
by pushing his ACKN STUD Switch 

• Student communications in progress indication - flashes 
when the student actuates a transmitter. The indication 
is removed when the instructor keys his microphone to 
speak over the ground transmitter of the . station which 
the student is attempting to reach 

• Malfunction indication - any one of the five malfunctions 
programed for a cockpit will flash when satisfaction of 
the programed condition is imminent 

• Automatic briefing or demonstration - indication that a 
briefing or demonstration is in progress 

• On-ground indication - weight on skids is indicated by 
WOS 

• Initial conditions established - RDY is displayed to indi- 
cate that the trainer is ready but the student has not 
assumed control 

• Parameter freeze - indication that one or more param- 
eters are frozen. 



The second example is provided by the mission display specified 
for device 2F101, T-2C Operational Flight Trainer (NTDC 1971). the dis- 
play format provides for two non-equal vertically aligned display areas on 
a CRT. The upper display area provides for situation displays; the lower 
display area presents supplementary information relative to the upper dis- 
play. The mission display provides the following display modes. 
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a. Cross-country display- -the area displayed is 120 x 120 
nautical miles . The upper portion of the display presents 

a graphic representation of all programed surface facilities. 
Each facility is identified with a three-letter code and the 
facility channel or frequency. A dotted line shows the air- 
craft ground track, and an aircraft symbol indicates present 
aircraft position. The bottom part of the display contains 
continuous tabular readouts for aircraft latitude, longitude, 
altitude, heading, ground-speed, DME reading of the BDHI, 
wind, and time from the start of the mission. The wind 
indication is graphically represented and the simulated air- 
craft ground track plot is the vector sum of the true heading 
horizontal airspeed vector and the wind velocity vector. . 
Four direction arrows are provided as an assist in slewing 
the display with a lightpen. Figure 21 is an example of a 
cross-country mission CRT display. 

b. Approach area display- -the approach area is 30 x 30 nautical 
miles. This display is automatically generated and replaces 
the cross-country display when the aircraft is within 30 
nautical miles of the surface facility in time. The center 

of the display is the in- tune surface facility. The represen- 
tation of the surface facilities, the simulated aircraft, and 
the aircraft track is the same as for the cross-country display. 
The bottom part of the display is also the same as for the 
cross-country display. Figure 22 is an example of the ap- 
proach area display. 

c. Ground Controlled Approach (GCA) display--an area of 10 
nautical miles from touchdown is presented. The upper part 
of the CRT is an elevation and azimuth display. Runway 
heading and glide slope elevation is that of the pertinent sur- 
face facility in the approach display mode. Aircraft position 
is represented by an aircraft symbol with its altitude and 
ground track represented by dashed lines on the az-el dis- 
plays. The lower display area on the CRT presents con- 
tinuous tabular readings for aircraft latitude, longitude, 
altitude, heading, ground speed, DME reading of the BDHI, 
wind, and time from the start of the mission. Wind indication 
is graphically represented. Instructions based on the air- 
craft position are displayed in this area. Figure 23 is an 
example of the GCA display. 

3.4.3.4 Controls for Display Selection and Modification. The design 
goal is to provide manual input means, compatible with the CRT displays, 
permitting communication with the computer with a minimum of intermediate 
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Figure 21 # Cross -Country Mission 
CRT Display. 



transformation of information. The controls must be: compatible with 
tabular and graphic formats, flexible) simple to operate, and conservative 
of console space. 

The console keyboards (alphanumeric and functions) and the 
lightpen are currently the most effective means for calling up, changing 
and modifying displays of student performance data and mission information 
and for providing the required control for instructional actions (see Chapter 
3.4.9). The instructor is provided the flexibility for selecting specific 
classes of data and initiating desired instruction pertinent to each student, 
as required. The controls for the instructor station display system must 
be capable of interacting with the CRT displays in all pertinent operating 
modes for all trainee stations. The specific information requirements for 
design are determined within the following control functions: 
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Figure 22 . Approach Area Display. 
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Figure 23. GCA Display. 
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select instructor display system modes involved 
in off-line and on-line operations (e.g. , DRED, 
PLAN, PROBLEM and CRITIQUE modes) 

select displays associated with monitoring, evalu- 
ating and scoring student performances in the 
PROBLEM MODE (e.g., error, performance and 
status information) 

select specific mission areas or events (e.g., mission 
displays, procedural performance (TMS)) 

modify preprogramed scenarios enroute during the 
exercise 

select and control the display of specific CRT pagina- 
tion (alphanumeric and graphic) 

initiate desired instructional actions during the exercise 
(e.g., reinstruction, demonstration) 

■ select hard copy records of student performance. 
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3.4.4 INFORMATION DISPLAY REQUIREMENTS FOR MONITORING, 
EVALUATING AND SCORING PERFORMANCE 

The specification of display requirements is largely dependent on 
the information requirements in terms of: content, amount, variety and 
speed of information change, complexity and the student-to-instructor ratios 
(i.e. , in multiple but independent trainee stations). The business of shaping 
student behaviors during a series of exercises requires careful thought 
about the manner in which student performance and system information is 
displayed and on the concomitant control capability. In short, an organized, 
unambiguous i fast access information format commensurate with the device 
under consideration is required. 

The emphasis is placed on achieving sufficient information to moni- 
tor and evaluate the significant aspects of any student's performance and 
system status at any time during the exercise. Thus, efficiency and flexi- 
bility in information presentation and updating, and in information selection 
is the desired design goal* Implied in this are the human engineering re- 
quirements in layout and usage patterns to minimize confusion, distraction 
and clutter in the displayed information. 

The information requirements outlined below provide a compre- 
hensive listing to account for most device classes and training situations. 
The human factors specialist need only consider that which is pertinent to 
the contemplated device. (This is in keeping with our intent to provide 
guidance and support to the human factors effort in developing the technical 
approaches to any device design.) The listing accounts for the most de- 
manding instructional situations and assumes the following: multi-format 
CRTs; multiple displays and multiple students undergoing simultaneous but 
independent instruction. The following information classes should be con- 
sidered. 

• Mission data--The information includes: a listing of the train- 
ing problems, the training problem number selected, active 
trainee stations (multiple, independent trainee stations) and 
the students assigned to a specific instructor/device operator. 

• Display of student history — This is information on the training 
history of each student and is displayed at the instructor station 
when the student is in the simulator (e.g. , provided by a card 
reader to read into the computer for data entry purpose). 

• Mission/ scenario area — This involves the display of mission, 
tactical environment, navigational areas (latitude, longitude) 
or relative geometry in the pertinent display modes. 



-148- 



NAVTRADEVCEN 71-C-0013-1 



. Own vehicle and support/ friendly vehicle display--This in- 
cludes identity, position, track (past history) . 

• Other mission events/installations — This includes fixed 
installations (bases, radio navigational facilities) or mission 
actions (e.g., radiating EW emitters, airborne interceptors, 
etc.). * 

• Target data 

- range, bearing, heading, altitude/ depth information 

- target contact/detection (onset, attenuation, disappear- 
ance as a function of range and environmental conditions) 

- tracks/position (differentiation of targets) 

• Total situation display (e. g. , tactical engagement, GCA, 
plot of total ground track with tolerance bands and check- 
points on which student track history is overlaid, etc.). 

• Environment- -(all pertinent parameters). 

• Console operating modes- -This refers to the display of the 
different modes of operation required to control the total 
training exercise and to monitor and evaluate student .per- 
formance (see Chapter 3.4.6). 

. Indication of system state or mode of operation and present 
phase or point in the mission or exercise. 

. Location (geographic) of all students in the exercise. 

• Variance of vehicles about the ideal track (e.g. , aircraft 
deviations from true ground track) . 

• Status of events initiated in the trainee station (e.g. , indi- 
cation of jammer activation, deployment of sonobuoys, 
chaff dispense, weapon selection and deployment, "fanfare" 
deployment, etc.). 

• Indication of each student/team control or tactical action 
and consequences of this action (e.g. , range scale settings, 
sector selection, operating mode selection, radio frequen- 
cies selection, etc.). 

. Indication (and time) of student/ team detection and identifica- 
tion of targets, from onset. 
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Indication of instructor actions, initiated and under current 
control or monitoring. 

Instructor control of vehicles — This refers to the information 
needed for controlling a submarine target or "flying" an air- 
craft (e.g., position, heading, altitude/ depth, speed, track 
of the pertinent vehicle). 

Performance information on each student or team — This refers 
to error/ status/ activity information (i.e. , immediate or time 
sampling by type, parameter or error class). 

Error alert indication of out-of-tolerance performance (e.g. , 
computer freeze; audio message to student). 

Indication of any/all vehicles out-of-tolerance (e.g. , bearing 
error, offset error, latitude and longitude error). 

Indication of adequacy of student procedural performance-- 
This refers to checklist procedures (normal and emergency) 
in sequence and in error (omission and commission). 

Indication of student/team error deviation from programed 
tolerances --This refers to information on performance which 
exceeds the error envelops set for the exercise (total error, 
errors per parameter, errors per class). 

Types and number of communications errors, 

Communications -navigation indications (e.g., prerecorded 
automatic ATC clearance given the student). 

Indication of the actions requested of a student — This refers to 
specific questions (messages) presented from a finite list (e.g. , 
specific stimuli (request for an activity) provided by an annun- 
ciator panel, and showing student response to the message and 
the resultant feedback of performance information). 

Indication of malfunctions/emergencies/failures (e.g., speci- 
fic types and locations, aircraft partial panel mode). 

Indication of all critical time and event happenings enroute 
during the exercise (selectable on demand). 

Visual display of student instrument panel status and control 
manipulations (e.g. , via CCTV). 
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Status information requests— This refers to the type and 
frequency of student requests for assistance relative to 
geographic position (where am I?), target deportment, etc. 
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3.4.5 CONTROL CAPABILITIES FOR MONITORING, EVALUATING 
AND SCORING PERFORMANCE 

A number of classes of controls, many of which are direct counter- 
parts to the display capability, serve to operate the training device and to 
enable the shaping of student behaviors. A capability is required at the 
instructor console to control the information displayed to the student as 
well as to control the information displayed about 'tudent and about system 
performance. 

Instructor console controls may be defined according to the func- 
tions performed, which in turn. influences decisions on control selection 
and on control placement in the console layout. Four groupings of controls 
can be identified. 

• Mission/instructional control (on-line) — This is the major 
grouping of controls. They enable the structuring and control 
of training during the exercise, and range from switches 
through control- display indicators to manual input devices. 

In multi-man trainers where fast access to much information 
in multi-formats is necessary, the selection of manual input 
controls compatible with the displays is a crucial determination 
for training management. (See Chapter 3.4.9.) 

. Scenario modification (off-line)— These are the controls for 
modifying or generating new scenarios where preprogramed 
training exercises are employed. Where automated prepro- 
gramed exercises are not warranted, the pertinent controls 
arc these employed in the manual setup of the initial conditions 
for an exercise. 

• Ancillary and power supply- -These are the controls associated 
with the operation of the training device, e.g. , power to the 
instructor console, exercise initiation, emergency stop, lamp 
test, etc. 

• Communications — These are standard controls, most often, 
replicas of existing operational equipment for two-way com- 
munications with: the student(s) (individually or grouped); the 
remote instructor(s); the training device operator; and other 
associated personnel. 

The following control capabilities should be accounted for, as 
appropriate, to the training device under consideration. 

• Device operation-- This involves controls for power, checks, 
system initialization, problem start, halt and override. 
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Trainee Station Controls 

- power 

- motion system activation and deactivation 
• playback of cojitrol performance 

- placement of aircraft under simulated co-pilot control 

- display of GCA command information (e.g. , on graphic 
plotter displays) 

- instruct hard copy printer 

Means for displaying student history (multiple, independent 
student context). 

Means for positioning units in an exercise, both initially at 
problem start and during tHe exercise* 

Environment (media) controls--e. g. , earth f s atmosphere, 
water conditions . 

Means for selecting mission/ scenario area or scale. 

Means for inserting and manipulating targets in an exercise 
(e.g. , time and position of entering targets, repositioning 
of targets, control of the range between targets or between 
own vehicle and target(s)). 

Means for inserting parameters and events at prescribed 
times in an exercise (manual insertion in scenarios not 
preprogramed 

Means for inserting non-programed events or eliminating 
programed events in preprogramed scenarios — This capability 
provides the instructor various options in developing instruc- 
tional strategies tailored to the individual student f s abilities. 
The instructor has the option to manually override the entry 
of a prescribed normal or emergency event when it is in 
position for entering the mission (for students who are below 
performance expectations). Additional normal and emergency 
events may be inserted into the preprogramed scenario to 
advance students at their best rate of progression (for students 
who exceed scenario requirements). 

Means for selecting CRT display modes and formats— This is 
a complex and crucial requirement and affects significantly the 
total instructional control capability.* Based on computer 
assistance in monitoring, evaluating and scoring performance, 



-153 



NAVTRADEVCEN 71-C-0013-1 



means are required for selecting and controlling pertinent 
training parameters from the complete training data on all 
students. The control design issues are the following: 

- format selection: alphanumeric and pictorial representation 

- display mode selection: a variety of operational modes are 
pertinent depending on the specific device requirements. 
Basically, however, the control capability subsumes the 
basic display modes of: 

problem setup 

monitoring 

error evaluation 

error alert (out-of-tolerance) 

scenario/problem modification. 

demonstration 

- selection of specific data: various aspects of performance 
information must be available. This includes the capability 
to select information on: 

individual/ all students 
system performance 
performance error data 

specific aspects of performance as a function of 
monitoring, evaluating, or scoring modes 

Means for varying task difficulty by freezing (holding constant) 
individual parameters in an exercise for a given student (e.g. , 
during initial instruction in a flight trainer, altitude may be 
held constant to enable the student to concentrate on heading 
control; axis lock, enabling the lock or release of one or more 
motion axes (roll, pitch, yaw)). 

Means for inserting/ denying failures or emergencies into an 
exercise. 

Means for direct access to the device computation system 
(e.g. , manual input controls such as keyboards, lightpen, 
etc . ) . 

Communications controls. 

Means for controlling vehicles--e.g . , controls for steering 
a submarine, "flying" an aircraft. 
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Controls to tally defined complex responses of students 
(e.g. , student responses to communications procedures, 
problem solutions). 

Means for recycling the system in automated preprogramed 
training exercises. 

- reinstructiom recycling a portion of the mission to direct 
an errant student to accomplish again that which he has 
just completed. 

- remedial branching: when student performance deviation 
or error rate exceeds the preset criterion for an error 
class/type, the student is automatically recycled to a 
remedial training segment. 

- capability to recycle students who exceed performance 
expectations to another segment more appropriate to their 
demonstrated skills. 

Means for initiating and terminating demonstrations, in real- 
time and in slow-time. 

Means for selecting performance information to be scored and 
recorded. 

ivlsans for inserting changes in the preset error tolerances 

this refers to the capability to change error tolerances (pre- 
programed increments) when student error per class exceeds 
the preset tolerances. 

Means for monitoring and controlling student procedural per- 
formance—This refers to the control in instruction in defined 
procedural sequences, e.g., checklist procedures (normal 
and emergency) in flight simulators . 

Means for selecting hard copy records of student performance, 
both enroute during the mission and after exercise completion , 
for critique purposes. 

Interlock controls between the instructor station and ttie train- 
ing device operator console— This concerns minimizing the 
possibility of an inadvertent or undesirable insertion of addi- 
tional parameters to a preprogramed scenario. For example, 
a key switch may be employed which must be activated at the 
TDO console prior to enabling any parameter insertions via 
controls at the instructor console. 
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3.4.6 INSTRUCTIONAL OPTIONS CAPABILITY 

A goal in instructor station design is to provide an inherent flexi- 
bility in operations sufficient to the development of instructional strategies 
pertinent to each student, consonant with the purpose >f the training device. 
The basic requirement is the specification of software programing and hard- 
ware capabilities which provide the relevant display and control modes and 
the instructional assists and guidance supports in the management of train- 
ing. In short, the instructional efficacy of the training system hinges on 
the availability of these kinds of features in design. 

This flexibility in instruction which takes advantage of the most 
recent training technology is accomplished in various ways . To facilitate 
disc ission, the varieties of instructional options are organized as follows: 

• modes of operation for monitoring and evaluating student 
performance 

• guidance supports 

• specific stimulus presentation 

• assists for controlling problem complexity 

• knowledge of results 
» mediation 

In most instances, programs are required for use with the simulation pro- 
m gram to enable instructor manipulation of instructional events and exercise 
parameters . The information requirements for achieving the capabilities 
defined in each of the above listed categories are described next* 

3.4.6.1 Modes of Operation for Monitoring and Evaluating Student 
Performance. Three basic trainer modes are utilized, defined in terms 
of instructor control and involvement. 

» automatic mode — This is predicated on an automated 
monitor, evaluation and scoring system and prepro- 
gramed training exercises. The automated training 
features include, demonstrations, problem sequencing, 
student performance and error recording, out-of- 
tolerance alerts and adaptive sequencing* Student 
performance is continually evaluated by the computer 
subsystem (comparison of actual with preset tolerances) 
and hard copy records of performance are available. 
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The instructor is not involved unless the student is 
unable to perform within expectations or requests 
assistance. This mode includes the evaluation 
(checkride) exercise employing a standard prepro- 
gramed exercise with automatic performance recording 
and error scoring for later off-line analysis with no 
instructor intervention unless the student is unable to 
continue. 



• semi-automatic mode—This is predicated on the 
automated instructional assists indicated above (i.e. , 
preprogramed scenarios, automated scoring and 
various automatic error alerting features), but is 
instructor-controlled to achieve the full expression 
of the manual adaptive capability.. 

• manual mode- -This is predicated on instructor involvement 
in structuring and controlling training and includes the 
manual installation (via controls and displays) of the initial 
conditions and the control of mission events via a script 

or other bases. Performance evaluation may be based 
on judgments about performance via displayed informa- 
tion and student actions and communications, with or 
without time and event recording of measurement 
information. 



Our emphasis is placed primarily on semi-automated trainer 
operations since the instructional options place the greatest demands on 
the display and control capabilities in interaction with the computer pro- 
graming. The information requirements for this mode center on the dis- 
played information and relevant controls for selecting, monitoring and 
evaluating all aspects of student performance and the means for controlling 
the entire trainer and the training exercise. The basic variations in the 
CRT displays) depend on the instructional functions and the information 
desired. These include; 

a. monitoring mode- -For determining the performance and 
mission status of each student irrespective of specific 
errors made or out-of-tolerance conditions. 

b. monitoring and error evaluation mode — Continual inter- 
rogation is made of specific performance information as 
the necessary input for decisions on training strategy, 
(i.e. , manual adaptive capability — the options include: 
continue as programed; freeze and rein$truct; freeze 
and demonstrate; increase/decrease problem difficulty 

by inserting/ omitting events in the preprogramed scenario). 
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c. error alert mode- -Special case of error evaluation 
where automatic freeze occurs when a student exceeds 
a preset error envelop (e.g., total or per parameter). 
Modified error tolerances (A error) must be established 
and inserted via appropriate controls into the computer. 

d. demonstration mode — Display of information demonstrat- 
ing to the student desired or ideal performance or aspects 
of this performance in real or slow-time (e.g. , maneuvers, 
instrument and control activations, specific EW emitter 
signatures, etc.). 

e. specific mission modes — Trainer mode and submodes 
particular to the operational counterpart. 

3.4.6. 1. 1 Display Requirements- -The following determinations must 
be made to achieve the display content and flexibility in data presentation: 

• Display format for each mode. 

• Types of display. 

• Events to portray per display. 

• CRT page formats. 

• Identification of the data common to each 
display mode. 

• Size of the mission area. 

• Information specific to the contemplated 
device. 

• Situational (graphic) display of total/ summary. 

(These requirements are described in detail in Chapter 3.4.3) 

3.4.6.1.2 Control Requirements- -The following requirements must be 
achieved: 

9 Controls for changing or modifying mission areas, 
CRT pages, operational modes. 

• Controls for calling up specific information classes, 
erasing information. 

• Controls for recycling portions of the displayed 
information. 

3.4.6. 1. 3 Instructor console modes of operation for an Air Navigation 
Trainer--An example of the information requirements for the operational 
modes proposed for Device 1D23 (Bark, et al 1969) is shown below. Basic 
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formats for visual presentation were defined to monitor and control the 
enroute exercise, control moving navigational facilities, monitor and assist 
students and evaluate student performance. This problem mode was divided 
into four submodesr 

♦ all aire raft- -This mode provides flight track and 
performance information on all forty students (except 
when operating as two distinct trainers, i.e. , twenty 
trainee stations per instructor console). 

• group aircraft-- This mode is identical to the all aircraft 
mode except that only ten trainee stations are repre- 
sented by the CRT presentation. Information pertinent 
to any trainee station which exceeds the established 
error tolerances, will blink on the CRT. 

• Aircraft out-of-tolerance--This mode presents only 
information on trainee stations that have exceeded the 
established limitations. 

♦ Specific aircraft- -This mode is identical to the aircraft 
out-of-tolerance mode except that information on a 
single trainee station is selectable at any given time. 

All tabular information presented during the problem mode common to the 
total training problem is common to the submodes and appears in the same 
formats and in the same locations. An example of the CRT display for the 
specific aircraft submode is shown in Figure 24. 

A keyboard control capability is provided for mode selection. 
A key is provided for each display mode and for selecting the scale of the 
geographic plot area. 

NOTE: In addition to the problem mode, additional 
modes are employed in off-line operations. These 
are: the daily readiness mode (DRD) for monitoring 
and controlling the checkout of the trainer; the prob- 
lem setup mode (PSU) for selecting or generating a 
preprogramed lesson from the trainer mass memory 
storage; the replay mode (RPL) which provides the 
means for replay of the exercise in real or fast time; 
and the inspect and change mode (IAC) which permits 
the instructor to inspect and change the library of 
instructor-controlled parameters . 
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3.4*6.2 Guidance Supports. A means for providing supplementary 
information about performance involves the use of cues and prompts in 
design. The value of these techniques which provide guidance during train- 
ing, is well recognized; it is the matter of implementation of these design 
options that is of interest here. Our concern includes both the design of 
supports provided to the student and the assistance provided the instructor 
to simplify and improve his instruction. 

So far as the student's progress is at issue, considerable data 
on learning have indicated that task relevant guidance (i.e. , stimulus 
support) is particularly useful during initial learning, for it emphasizes 
the relevant aspects of performance, limits exploration time and minimizes 
gross errors in performance. As learning progresses, these supports are 
eventually withdrawn until the student responds to the stimuli found% 
.terminal performance. The information requirements for providing vari- 
ous means of guidance support are described next. 

3.4.6.2. 1 Demonstration- -This refers to the capability to provide the 
student preprogramed examples of training problem solutions that must be 
mastered. Examples are provided to demonstrate desired performance of 
selected system behavior patterns which may be used at any point in the 
exercise (during briefing or enroute in the problem) as dictated by the 
mission requirements and by the adequacy of student performance. The 
pertinent content covers a broad range, including: instrument and control 
activations of idealized maneuvers of an aircraft; preprogramed signatures 
of classes of EW emitter signals and characteristics of preprogramed 
emitters; emergency or malfunction conditions; a ground-controlled approach 
(GCA) with voice commentary; and specific control requirements in a task 
or job segment. 

The information requirements for providing this design capa- 
bility include the following: 

. definition of the specific demonstration events 

required and the total number that will be programed. 

• definition of the specific content and length of each 
demonstration event- -This includes the initial condi- 
tions and prescribed dynamic data (speed, rates, rate 
changes) to execute the defined problem solution or 
maneuver with synchronized verbal commentary (as 
appropriate), or the series of points or the patterns 
in sequence that define the desired display to the 
student. 

• specification of the tolerances in each problem solution 
to provide the desired fidelity in time and movement, 
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i.e. , technical accuracy commensurate with the 
training objectives (idealized/ school solution; 
desired sequence! etc*)* 

• specification of where in the mission scenario 
preprogramed demonstrations are desired. 

• specification of performance conditions (e.g. , 
error tolerances exceeded by the student) which 
initiate automatically the demonstration required 
to assist the student to perform within tolerances. 

• specification of the performance data display(s) 
and controls to enable the instructor to select 
and initiate a defined preprogramed demonstra- 
tion (as required in tailoring the instructional 
strategy for a given student). 

• controls for slow- or real-time demonstration; 
for halting the demonstration in progress; and 

to access a portion of the demonstration sequence. 

• capability for generating new or modified demonstra- 
tion sequences (off-line). 

An example of the capability for automated demonstrations 
of training problem solutions is provided by Device 2B24, Synthetic Flight 
Trainer System. A library of demonstrations (10 problem solutions) is ' 
provided. Each demonstration contains the initial condition data and other 
data necessary to execute the demonstration flight. These data are retained 
in auxiliary storage but are accessible during a training session without 
instructor intervention. Upon request by the student or by the instructor, 
each demonstration is executed in such a manner that the cockpit instru- 
ments and the motion module exhibit the movements which would be ex- 
perienced if the demonstration were actually being flown. A synchronized 
pre-recorded verbal explanation and commentary is provided for each 
example. The demonstration is provided in real-time or in slow-time 
(one unit of real-time equals two units of slow-time) or in some combina- 
tion of real-time and slow-time as appropriate for student understanding 
of the problem being demonstrated. Emphasis is placed upon instructional 
quality as well as on technical accuracy of the demonstration. The follow- 
ing demonstrations are programed. 

• The interpretation of attitude and flight control 
instruments and relationships between displayed 
information and control movements. 
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• VOR approach, including reporting procedures and 
execution of a missed approach* 

• A tactical ADF approach, including reporting procedures 
and execution of a missed approach. 

• A standard ADFapproach, including reporting procedures, 
wind drift correction, and execution of a missed approach* 

• An ILS approach, including wind-drift correction, report- 
ing procedures, and execution of a missed approach* 

• Control of the aircraft during climbs, descents, turns, 
climbing and descending turns and level flight by refer- 
ence to attitude instruments* 

• Interpretation of navigation instruments. 

• Holding at an intersection, including entry from 90° 
off-axis, wind-drift correction and departure at a pre- 
determined time* 

• An in-flight engine failure and air restart. 



♦ An instrument takeoff. 



An example of a demonstration segment ?s shown in Table 10*. 
It depicts an instrument takeoff (ITO) for a helicopter* and involves the 
briefing, the complete ITO, and the ITO in slow- time. 

3*4.6*2.2 Reins truction. Consistent with the manual adaptive capa- 
bility is the reinstruction mode* This enables the instructor to return to 
an earlier portion of a mission for any or all students who perform below 
expectations on the given exercise* The reinstruction requires that the 
student perform again that which he has just completed (e* g. , a segment, 
a flight leg or portion of a leg)* To accomplish this, the instructor requires 
the following (assuming the automated scoring capability): 

• dynamic display (CRT) of student performance 
information in all PROBLEM modes (alpha- 
numeric and graphic formats) 



This work involving the HH-52A helicopter was accomplished by the 
Human Resources Research Organization for the Coast Guard (see for 
example, Hall, Caro, Jolley and Brown 1969)* 
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• display of error data and out-of-tolerance data 
(exceeding preset tolerances) for all students 

• controls to halt the exercise for the given student 
and recycle the mission to the desired segment. 

This provides a means for controlling problem difficulty 
consistent with the student's demonstrated ability. In the same sense, the 
instructor has the option to nullify certain events in a preprogramed scen- 
ario (e.g. , eliminate a malfunction already programed, eliminate a. pro- 
gramed airborne interceptor attack in an EW mission) for students per- 
forming poorly. He may also insert new events in addition to those in the 
preprogramed scenario for the student whose performance exceeds the 
mission requirements. This provides for a controlled increase in problem 
difficulty for the purpose of keeping each student at the threshold of his 
ability during the training program . - » 

Difficulty level may also be under computer control. In this 
case, performance measures are continuously compared with preset stand-" 
ards, and when student error rate exceeds the defined tolerances for an 
error type or other grouping, the problem is halted and the student is 
branched automatically to a remedial training segment. The branching 
decisions and the rules for resumption of the standard exercise must be 
programed into the computer. 

3.4.6.3 Specific Stimulus Presentation. In complex training environ- 
ments, stimuli associated with sequences of performance are often vague 
and not easily identified. This hampers the control of the instructional 
process. There are instances, however, where specific stimuli can be 
controlled to advantage in the shaping of student behaviors. In these situa- 
tions, the student is presented specific stimuli at prescribed times which 
call for observable finite responses, for example, computations based on 
the present system state, geographic position determinations, or more 
generally, the performance of tasks where: the antecedent conditions are 
specifiable, the time of request is specific and the response completion is 
observable, definable and scorable.* 

The requirements for presenting and monitoring specific stimuli 
to the student involve the following: 

• trainee station display — Alphanumeric messages are 
are presented at programed times calling for a specific 
response from a student. 

• trainee station controls — Means must be provided (e.g. , 
keyboard) for the student to respond to a specific mes- 
sage or question. 
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• a list of messages to be used in conjunction with the 
lesson plan is provided. This includes the specific 
content requirements, message length and where in 
the exercise the message is presented. 

• instructor station display of the total list of messages. 

• instructor station display of the specific message 
presented, the student response and an indication of 
knowledge of results presented. 

• immediate supplemental information presented to the 
student about the results of his performance. 

• instructor station controls to call up a programed 
message — this refers to instructor-initiated messages 
at times different than programed in the lesson plan. 

• student response is automatically compared to the 
actual situation — the information requirements for pro- 
graming include, system status information, vehicle 
position movement and heading, rates of resources 
expended, etc. 

This capability for stimulus control is exemplified in device 1D23, whereby 
an annunciator panel in the trainee cockpit presents messages which call for 
specific behaviors from the student (see paragraph 2.3.3.2). 

3.4.6.4 Assists for Problem Control. A number of automated features 
can be incorporated into instructor station design to provide assistance in 
controlling problexpi complexity and in unburdening the instructor of certain 
monitoring and controlling functions in developing instructional strategies 
for all. students. These design options are outlined next. 

3.4.6.4. 1 Automatic Error Alerts. When a student exceeds a prepro- 
gramed errc- envelope in a training exercise (in terms of the scoring criteria 
established for the lesson number), a display of this information is provided' 
the instructor and a computer -initiated freeze is the outcome. The perform- 
ance alert indication is displayed alphanumeric ally on the CRT when, for 
example, one error less than the amount preset (e. g. , total errors, error 
class, or error on a given parameter) is accrued. The instructor options 
are, manual override of the freeze, or accept the freeze with procedures 
to continue. Controls are required to set in new error tolerance envelopes 
to continue the exercise. The information requirements include: 

• initial scoring criteria for the- exercise 
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• the classes/types of errors to be monitored and 
scored automatically 

• decision rules for the inception of the error alert 
condition 

• continuous alphanumeric display of error information 

• new error scoring criteria (A error change when 
preset tolerances are exceeded) 

• controls for inserting new error criteria (A error 
change). 

A variation of the automatic error alert mode is provided in 
device 2F100 (A-7E aircraft). The instructor is able to select ten prepro- 
gramed parameters for scoring (from thirty-one possible parameters), each 
with a defined reference performance criterion (one of five predetermined 
ranges). The program calculates the actual tolerances for the selected 
parameters and sets linkages in the simulation programs which trigger 
indications whenever these tolerance limits are exceeded. Whenever a 
parameter exceeds its tolerance limit or returns to in-tolerance, the pro- 
gram prints the following data. 

• parameter name , 

• aircraft position (latitude, longitude) 

• elapsed time 

• direction of the deviation 

• reference criterion value r 

3.4.6.4.2 Individual Parameter Freeze. It is desirable to provide the 
capability to freeze individual parameters for a given trainee station. For 
example, in flight simulation, the freezing of the altitude parameter enables 
the novice pilot trainee to concentrate on heading control. Failure to pro- 
vide this option may result in the breakdown of performance due to task 
overloading. The instructor station requires the controls to achieve this 
and in indication of the parameter freeze on the CRT display. Device 2B24, 
for example, provides for the following parameter freezes for any trainee 
station: altitude; attitude; heading; velocity; rate of climb/descent; rotor 
RPM; torque; turn rate. 

3.4.6.4.3 Autopilot Program. A desirable option f6r flight simulators 
is the provision of an autopilot program. This program automatically com- 
putes ideal or desired vehicle position, rates and rates of change for any 
programed maneuver or maintains a desired steady state of the vehicle. 
Such a capability is useful in early phases of training to control task loading, 
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enabling the student to concentrate on other tasks (e.g., comm/nav pro- 
cedures). The program is also used as a standard against whicft student 
performance can be compared and serves in the development of performance 
scoring values. Co-pilot simulation is provided in device 2B24 to enable the 
student to perform tasks incompatible with the control of the helicopter. An 
indicator button at the instructor station activates this capability, including 
an audio message to the student, via intercom, stating that the aircraft is 
under co-pilot control. The simulated aircraft flies within the programed 
tolerances in effect at the time of autopilot initiation, or continues in the 
steady state condition prevailing at the time of initiation. The same con- 
trCi including an audio message is used to return the device to student control. 

3.4.6.4.4 Additional Supports. A closed-circuit television (CCTV) sys- 
tem provides supplemental information for instructional control. A CCTV 
camera mounted in each trainee station affords the instructor (via a TV 
monitor) the opportunity to view student control manipulation, the instru- 
ment panels and displays (e.g., engine and attitude instruments in flight 
simulators, signal signatures in EW trainers) and, where pertinent, the 
correlation of these with an "outside" visual frame of reference (e.g., 
horizon). The CCTV capability is particularly useful in multi - student train- 
ing situations involving independent performance where the instructor must 
time-share among students. The information requirements for this capa- 
bility include: 

• decision on the use of a CCTV monitor in the 
trainee station 

• placement of TV cameras in trainee station(s) 

• selection of the type and number of TV monitor(s) 
in the instructor station 

• controls for operating the CCTV system (e.g. , 
camera selection, camera adjustment and position- 
ing, lens adjustment and zoom, selection of the 
monitor(s) upon which the image is displayed, ad-, 
justment of the monitor(s), and display to student(s) 
(e.g., split lens switches) 

• video recording and playback controls. 

A video tape recorder provides the playback capability. This unit (with a 
defined time capacity) accepts the output of a selected trainee station and 
can be used to play back the video record to any combination of trainee 
stations and to thfe instructor station via the TV monitors. 
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In, multi- student trainers, a student data device (e.g. , a 
punched card reader) may be provided in each trainee station to read into 
the computer the student's identifying and training history data. This in- 
formation is displayed at the instructor station CRT and is used as data 
entry information in the automatic selection of a training exercise tailored 
to the given student's level of achievement. The design requirements include 
the content and format of the input (e.g. , punched card) and the means for 
accepting the data for display and for automatic problem selection. 

3.4.6.4.5 Knowledge of Results. It should be made explicit that supple- 
mentary information about performance provided by equipment or by the 
computer is a desirable design option and should be implemented whenever 
feasible. In essence, the feature for design is that providing subsidiary 
(supplemental, trend) information about performance, 1) enhances student 
learning and performance, and 2) enhances the monitoring and control 
capability of the instructor. 

In the case of the trainee station, the design goal is to pro- 
vide augmented feedback of performance information via trainer equipment. 
This is a special case of information presentation in 'which the student is 
provided a signal immediately after responding which indicates how his 
performance conforms to expectations. Specific design options have J already 
been discussed in paragraph 2.4.4.?, of this. report. 

Supplementary signals or information have utility also in the 
instructor station design. Where feasible-, the following information re- 
quirements are appropriate in technical approach determination. 

• alerting signals--this refers to indications that 
student performance error has exceeded defined 
criterion limits, or that procedural error has 
occurred. In the first instance, a continuous 
display of error information (e.g. , alphanumeric 
format per defined error parameters or classes on 
a CRT) is provided together with an indication of 
status, such as a histogram depicting current level 
of achievement or as a qualitative change in pre- 
sentation as out-of- tolerance is approached. An 
example of the second instance is the blinking 
effect indicating a procedural error of omission 

or commission on the trainee monitoring system 
display (see paragraph 3.4.8.2). 

• . trend Information- -this refers to indications that 

a change in status is imminent, for example, a 
student is approaching an out-of-tolerance condition, 
and an instructional strategy decision is required 
from the instructor. 
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• summary information — this refers to supplementary 
information on defined, critical aspects or status of 
performance relevant to the control of instruction. 

• static pictorial representations --this refers to the 
reconstruction (e.g. , CRT display) of a segment or 
the entire sequence of a mission accomplished by a 
student* 

• feedback of overall performance- -this involves hard- 
copy printouts of each student's total performance (in 
desired formats), available from the computer, 

3.4,6*5 Mediation* A valid.design alternative concerns the selection of 
non-equipment options in the technical approach to the device configuration. 
The term, mediation, denotes this design approach which attempts to reduce 
or simplify the device hardware and substitute the instructor as an integral 
part of the training loop. Two aspects are subsumed under this concept: 
reduction (exclusion) of hardware solutions; and equipment simplification. 
The value of this approach is obvious in a variety of situations, generally, 
where instructional support and the information required for performance 
is quite diverse, depending on student performance (e.g. , communications 
links, ground- controlled approaches (GCA)). There are also more subtle 
applications where this approach is viable. Suffice it to say, the skillful 
use of these surrogates for actual equipments may improve the instructional 
capability by sharpening the focus on the purpose and the objectives of training 
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3.4,7 MEASUREMENT SYSTEM DESIGN 

The organizing feature in instructor station design as conceptualized 
in this report is the measurement capability. Knowledge about the outputs of 
performance provides a basis for assessing the effects of training and for 
predicting performance under future conditions. In short, performance 
measurement is the bridge between the student outputs and the information 
requirements at the instructor station. 

Many of the instructional functions described in this section on 
instructor station design are predicated on an automated monitoring, evalu- 
ation and scoring capability. Our discussion centers on automatic human 
performance monitoring and recording (computer programing and scoring 
equipment) for providing performance information that can be organized, 
manipulated and applied to achieve the purpose of instruction. Computer- 
aided performance measurement is predicated on a number of provisions: 

• Continuous automatic evaluation and scoring of each 
student 1 s performance. 

• Preprogramed mission scenarios or lesson plans.. 

9 Measurement routines incorporated in the simulation 

program to record pertinent events (e.g. , parameter deviations) 
from the preprogramed scenarios. (Automatic data recording 
pertains only to those situations where specific parameters or 
events and their tolerance envelopes can be established; 
the instructor, however, is directly involved in entering 
data for those situations where no tolerances can be 
established quantitatively). 

• Storage of measures for real-time monitoring, post-exercise 
printout, and end-of-course program evaluation. 

• Evaluation and scoring information available to the instructor 
via console CRT display and hard copy printout. 

• Capability for instructor selection of parameters or events 
for recording in addition to those inherent in the lesson, 
plan routines. 

A number of design decisions must be integrated to achieve an automated 
monitoring, evaluation and scoring capability. The human factors design 
issues center on the information requirements involved in the selection of 
measures, the organization of measurement information, and the display and 
recording of scoring information. The details that must be resolved are 
examined next. 
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3.4.7. 1 Selection of What to Measure. Decisions must be made in select- 
ing manageable units of performance, resulting in a listing of things worthy 
of measurement. Obviously, the types and arrays of measurement possi- 
bilities depend on the class of training device and on the training objectives 
and task structure. This governs the selection of the critical performances 
for scoring, i.e. , emphasis on manual control of vehicles > procedures 
following, signal detection, decision making, or coordinated team perform- 
ance. It is assumed that an outline of the task structure and the task anal- 
ysis has already been accomplished intteM.C. documentation and it is now 
mostly a matter of correlating skills and knowledge requirements with scores 
that are relevant and obtainable. 

The parameters and events to be sampled describe the interrela- 
tions among own vehicle, the target(s) and the media. The parameters re- 
corded relate to classes of data specific to types of training devices. For 
example, vehicle management parameters are of most importance to flight 
trainers whereas tactical employment and load (e.g. , vehicle units in an 
engagement) parameters are of significance to tactical team training devices. 
The relevant parameters reside in the following classes of data. 

a. Vehicle management — This class of data refers to vehicle 
positioning and power plant management parameters. It is 
of particular importance to flight simulators. For example, 
the eligible parameters for recording of aircraft perform- 
ance include the following: 



Fixed Wing - 

Heading 
Altitude 
Airspeed 
Pitch angle 
Roll angle 
G loading 

Stick longitudinal position 
Stick, lateral position 
Rudder position 
Brake pedal position 
Trim position 
Time 

Vertical velocity 
Turn rate 

Longitudinal stick rate 
Lateral stick rate 
Rudder rate 
Pitch rate 



Rotary Wing 

Heading 
Altitude 
Airspeed 
Pitch angle 
Roll angle 
G loading 

Cyclic stick longitudinal position 

Cyclic stick lateral position 

Collective stick position 

Rudder position * 

Brake pedal position 

Trim position 

Time 

Vertical velocity 
Turn rate 

Cyclic stick longitudinal rate 
Cyclic stick lateral rate 
Collective stick rate 
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Fixed Wing 



.Roll rate 
Yaw rate 

Engine instruments 

RPM 

Torque 

Flight path 

Flight path deviation 

Approach slope 

Approach slope deviation 



b. Relative geometry considerations — This class of data is con- 
cerned with own- vehicle position and action direction or with an emplaced 
unit (e.g. , carrier aircraft controller) in relation to other units/vehicles 
in an area of engagement. For example, in surface or subsurface vehicles 
the important parameters include the variations in range and bearing data 
(course, speed, depth maintenance or change), and rate of change of these 
relationships (closing, opening rates, etc.). For a fixed or emplaced unit 
(GCA or carrier control of aircraft) the parameters of concern include the 
specific aircraft in relation to the recovery site and the positions and move- 
ments among aircraft in a defined pattern. 

c Tactical employment of vehicle — These data refer to the rele 
vant parameters in the basic detection-to- engagement paradigm, involving 
own vehicle, friendly unit(s) and target vehicle(s). For example, in surface 
ship ASW training, the relevant parameters inhere in sonar detection and 
classification, in attack director operations involved in developing the fire 
control solution, and in certain Combat Information Center operations (e.g. 
CIC plots) in support of operating the ship as a fighting unit. 

d. Load — These data relate to the amount of "clutter" (i.e. , 
number of things) involved in a training situation, and include, the number 
of different units in an engagement (support units, friendly units, targets), 
command relationships (HUK forces, SAUs, etc.), and communications 
requirements . 

e. Weapons employment— These data refer to the parameters 
recorded on weapon selection and preparation and on relevant weapons 
actions (e.g., water splash point for surface to subsurface missile, torpedo 
launch and course; countermeasures). 

3.4.7.2 Selection of Measures. The measures selected are based on 
the kind of information desired about the performance. Usually, well- stated 
training requirements suggest quite easily the corresponding measures and 



Rotary Wing 

Rudder rate 
Pitch rate. 
Roll rate 
Yaw rate 

Engine instruments 

Rotor RPM 

Torque 

Flight path 

Flight path deviation 

Approach slope 

Approach slope deviation 
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specific performance criteria. One of the problems in programing the 
computational system is determining what aspects of performance axe to 
be included in the measure. A logical progression is to correlate desired 
measures and scores with relevant portions of the mission profile. An 
idea for such a framework is shown in Table 11 which depicts a series of 
mission-phased measures and appropriate scores for an airborne Electronic 
Warfare Basic Reconnaissance mission. In addition, some examples from 
training devices of recent vintage serve to further exemplify the options 
available. 

The design of Device 2-F100 (NTDC 1970) specifies a total of 
thirty-one parameters to be scored with the means provided the instructor 
to select no less than ten parameters to be recorded simultaneously for 
performance evaluation. The total list of parameters is shown below. 



Heading 
Altitude 
Airspeed 
Pitch Angle 
Roll Angle 
G Loading 

Longitudinal Stick Position 
Lateral Stick Position 
Rudder Position 
Throttle Position 
Flaps 

Landing Gear 
Speed Brakes 
Thrust Attenuator 
Elevator Tab Up 



• Elevator Tab Down 

• Time 

• Vertical Velocity 

• Turn Rate 

• Longitudinal Stick Rate 

• Lateral Stick Rate 

• Rudder Rate 

• Pitch Rate 

• Roll Rate 

• Yaw Rate 

• . Engine Instruments 

• Canopy Controls 

• Seat Controls 

. Flight Path and Flight 
Path Deviation 

• Approach Slope and Approach 
Slope Deviation 

. RPM 



An organization of these relevant measurement parameters 
appropriate to the phases in a mission profile is suggested in Table 12 
for a mission sequence involving the following: Originate Airfield A, nor- 
mal takeoff; fly over VOR TAC radio station; proceed on radar navigation 
leg with etiroute malfunctions/emergencies; conduct GCA; normal landing. 

For the Air Navigation Trainer, Device 1D23 (NTDC 1971), 
measurement is conducted for critical navigation parameters and events 
as defined in the preprogramed lesson plans. Objective measures are ob- 
tained as follows: 



• Number and duration of tolerance violations by type. 

# Bearing and offset error. 
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TABLE 11. BASIC EW RECONNAISSANCE MISSION MEASURES 
AND SCORES 



Mission Phase 


Measure 


Scores 


Flight planning 


N/A 


N/A 


Pre- flight setup. 


Switch set-up positions 

• 


Sequence of switch positions 
(checklist format) 

Switch setting error 

Omission of switch setting 

Malfunction correction 

Communications (position of 
channel switches) 




Equipment preset 

(bands/frequencies 

requirements) 


Switch positions in equipment 
preset 




Swi etup per signal 


Switch position in search 
(signal number worked) 

Time to 'detect (onset/signal 
to equipment tuned for analysis) 


- 


— 


Cue buildup (onset of signal to 
search action) 

Switch position in identifica- 
tion (wrong emitter) 

Time to identify 

Signal centering switch 
positioning 

Time to complete 




Signal setup for DF 
bearing 


Switch position (error) 

Search switch 
Receiver bandwidth 
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TABLE 11. BASIC RECONNAISSANCE 
AND SCORES (Continued) 



MISSION MEASURES 



Mission Phase 


Measure 


Scores 


* 




Attenuator setting, bandwidth 
path 






Auieiuia selection |Consiueri ng 
polarization tuning) 






ALA-6 cursor position relative 

to Actual twfcaWtior 


- 




Time to initiation of DI 
bearing 




• 


Time to complete 






Re- analyzing the same signal 






fllO hi it 1 Hurt Kaftiw\A« — * ■ * 

Ksw; uuxiuup Dctween signals 




- 


Number of signals analyzed 




Recording 


Time' of recording initiation 




* 


Switch position error 
- wafer switch 
antenna selection 
audio/recorder level 
recorder operative 

Correctness of recording 
(error) 




Analysis 


Log keeping (off-line, post- 
mission) 




Subsequent DF bearing 


Time and/or bearing change 
between DF bearings 

Switch and cursor position 
error 

Number of DF bearings made 
per signal 

Total time per signal 
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TABLE 11. BASIC RECONNAISSANCE MISSION MEASURES 
AND SCORES (Continued) 



Mission Phase 


Measure 


Scores 




Priority search 


Signal attended to per time 
switch position error 


- 


Threat priority 


Time of handling new, higher 
priority threat . 

Switch positions 




General area 
reconnaissance 

+* 


Failure to detect/analyze 
signal 

Total time used per signal 


- 


Evasive maneuvers 


Maneuver selection 
Time of initiation 




Malfunction detection/ 
correction 


malfunction 
Malfunction correction 




Communications 


Switch positions on interphone 
and communications equipments 


Post-flight shutdown 


Switch shut-down positions 


Sequence of switch positions 




c 


Switch setting error 

Communications (position of 
channel switches) 
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TABLE 12. SIMULATOR MISSION EXAMPLE 



Defined Task Sequences 
in Mission Profile 


Relevant Measures 


Pre-star!; cockpit procedure 


Procedure error, trainee monitor 
system (TMS) 


Engine start 


Procedure error (TMS) 


Pre-takeoff procedures 


Procedure error (TMS) 


Takeoff 


Aircraft attitude, geometry, force/ 
rate parameters (turn rate, longi- 
tudinal and lateral stick rates vertical 
loading, rudder rate); discrete events 
■ (flaps, gear, elevator tabs, power) 


Navigation leg (site A to site B) 




Monitor Flight envelope and 
defined maneuvers 


Aircraft attitude, geometry, force/ 
rate parameters (as pertinent to 
defined maneuvers); time (to initiate 
events, to complete event* overall 
time to activity completion) 


Engine fire (emergency) 


' Procedure error (TMS) 


PCI system failure (emergency) 


Procedure error (TMS) 


Wing fuel transfer 


Procedure error (TMS) 


Navigation leg (site B to site C) 




Monitor flight envelope and 
defined maneuvers 


Aircraft attitude, geometry, force/ 
rate parameters (as pertinent to 
defined maneuvers); time (to initiate 
event, to complete event, overall 
time to activity completion) 


In-flight control malfunction 
(e.g., auto pilot) 


Aircraft attitude, geometry, force/ 
rate parameters (stick rates, rudder, 
roll, yaw and pitch rates, G-loading); 
Position of aircraft controls 
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TABLE 12. SIMULATOR MISSION EXAMPLE (Continued) 



Defined Task Sequences 
in Mission Profile 


r 

Relevant Measures 


Engine control 


Aircraft attitude, geometry, force/ 
rate parameters; position of aircraft 
controls; engine instruments 


Flame out 


Procedure. error (TMS) 


Surface controls malfunction 


Aircraft attitude, geometry, force/ 
rate parameters; position of. aire raft 
controls 


Flight instrument failure (e.g. , 
ADI, roll channel) 


Aircraft attitude, geometry force/ 
rate parameters; position of aircraft 
controls; speed brakes; engine 
instrument? 


ground controlled approach 


Airspeed, flight path, approach slope! 
vertical velocity, throttle^ position, 
rudder* Ditch rates, elevator- tab* 
RPM, engine instruments 


Pre-landing check list 


Procedure error (TMS) 


Landing 


Aircraft attitude, geometry, force/ 
rate parameters 


After landing checklist 


Procedure error (TMS) 


Before engine shutdown checklist 


Procedure error (TMS) 
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• Communication errors. 

• Flight change command and flight change accuracy. 
. Number of position requests. 

• Annunciator-response accuracy and latency (see paragraph 
2.3.3.2). 

• Variance expressed in standard deviations and percentiles 
for integrated time and distance about true ground track, 
time off true ground track, and number of deviations from 
course. The standard deviations and percentiles reflect 
the students performance relative to the ideal criteria arid 
relative standing with other students at the same point in a 
preprogramed lesson computed accumulatively every half 
hour. 

For the Synthetic Flight Training System (Device 2B24), the 
performance parameters recorded during the automatic (non-adaptive) mode 
of operation include: • 

• Airspeed 
. Altitude 

• Flight path 

• Rate of climb 
. Rate of turn 

. Pitch attitude ' 

• Bank attitude 

• Rotor RPM 

• Gas produced RPM 
. HEGT 

In addition, the frequency of procedural error in aircraft control tasks and 
errors in communication (e.g. , tuning the wrong frequency, error in the 
content of a radio transmission, etc.) is recorded by the instructor. An 
error button is used to record the time of occurrence of each error per- 
ceived by the instructor. 

3.4.7.3 Establishing Performance Standards. Various means are avail- 
able for achieving performance standards. Standards can be set for a number 
of performances from the rational analysis of the system and task structure. 
For certain behaviors, adequacy of performance may be differentiated on the 
basis of time to perform. Reaction time may be critical or error response 
may be detrimental unless corrected (assuming adequate feedback of per- 
formance information) within a short time period after the error response. 
In other instances, emphasis may be placed on quality and precision of 
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response or on the integration of activities that indicate an ability to cope 
with a range or unforeseen contingencies. 

Another approach to the development of interim standards in- 
volves the use of the a priori model of system and vehicle performance 
which is developed during the design phases of the operational system. 
The relationships between components as specified in the model can at 
times be translated into terms allowing a quantitative expression of per- 
formance levels required for successful performance. 

For more complex performances, the criteria should be based 
on empirical evidence. Ideally, experiments and actual data collection in 
the operational setting are desired for establishing the standards. ' Prac- 
tically, however, expert opinions are used to es.tablish criteria for success- 
ful performance. This intuitive approach is good where standards do not 
exist or have not been quantified. Interim criteria can be developed in this 
manner since subject matter experts can provide solid guidance in the 
development of standards. Also, obtaining repeated scores will allow the 
development of criteria. For example, continual measurement in the on- 
line simulator sharpens the ability to specify required performance levels. 
The error criteria should be flexible to account for stage of training, and 
modifiable to enable change as experience with the training program is 
gained. Thus, initially, tolerances are set intuitively based on previous 
experience and judgment and are refined as a data base accumulates with 
experience in the use of the system or as training objectives are modified. 

Quite often, useful performance tolerances have been established 
by the using agency based on much actual experience. This has been parti- 
cularly the. case with flight simulators and the fashion has been to accept 
performance parameter tolerances based on this expertise. For example, 
the performance tolerances for the automatic adaptive training mode in 
Device 2B24 have been set in this way (Table 13). 

In other instances, automated evaluation and scoring provides 
a comparison of student performance with an ideal performance model 
(with or without a weighting of performance factors). Performance cri- 
teria for Device 1D23, foj^xample, are developed from mathematical 
models of ideal navigator performance represented in the preprogramed 
lesson plans. The models include a performance corridor in X, Y and Z 
coordinates and consist of the basic parameter values of the lesson plans, 
to which constants can be added and varied. 
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TABLE 13. PERFORMANCE TOLERANCES FOR ADAPTIVE 
SEQUENCING IN DEVICE 2B24 



Performance Parameter 


Acceptable Tolerances 




Altitude 


t 100 feet 




Airspeed 


± 10 knots 




Heading 


f 5* 




Vertical Speed 


t 200 fpm 




Rate of Turn 


i lo/sec 




Ball Position 


f 1/4 ball 




Pitch Attitude 


i 40 




Roll Attitude 


±5<> 




Course Deviation 


i 1 dot 




Glide Slope Deviation 


* 1 dot 




Torque Pressure 


f 4 psi 





3.4.7.3. 1 Modifying performance criteria--Consideration must be 
given to the capability for modifying the computer scoring criteria in the 
software programs. In the case of an error alert whereby the computer 
halts the problem when a student exceeds the permissable error envelop 
in terms of the preset scoring criteria established for a particular mission 
or stage of training, the requirement is to set in new error criteria to es- 
tablish the next: error alert (computer halt) level. This can be accomplished 
via CRT display (e.g. , page(s) of alphanumeric information for error anal- 
ysis) and appropriate controls (e.g., keyboard, lightpen) to address the 
computer to override previous error criteria and modify (as required) by 
inserting new tolerances. This enables the continuation of the exercise 
according to the training strategy elected by the instructor. 

In Device 2F100 (A7-E aircraft), the instructor has the capa- 
bility to preprogram five different values and ranges of difference for each 
parameter (i.e. , specifying different values for differing problem segments). 
In addition, the tolerance envelopes for each parameter can be modified 
(five levels of A error) at the console with the problem underway. 

3.4.7.4 Sampling of Performance. A major concern in measurement is 
the quantitative determination of how student performance compares to the 
ideal pr the desired performance for a given circumstance. A most effec- 
tive means of scoring is to record out-of-tolerance performance, i.e # . 
error deviations in excess of a preset value or envelop. Measurement 
routines can be incorporated in the simulation program to record param- 
eter deviations from mission parameter profiles (this is the scoring con- 
vention in Hight simulator*). Additional consideration should be given to 
the following capabilities: 

i 
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scoring of procedural performance in terms of ^ 
errors in sequence and errors of omission in a 
preset sequence (see paragraph 3.4.8.2). 

recording the performance of critical events (and' 
time of performance) in the lessor plan. 

use of the instructor as a link to the computer for 
scoring behaviors not easily processed automatic- 
ally (e.g., communications). 

Data sampling rates— This design consideration concerns 
such issues as: how often the measurement data are sampled in the training 
exercise; the length of time each sample is examined; and the places in the 
mission scenario wherein the sampling is conducted. These scoring deci- 
sions are correlated with computer "space availability;" Logically, the 
sampling time for, defined tasks (e.g. , a 90° level turn at a rate of 3°/ sec) 
is set to include that complete performance; for more or less continuous 
controlling (e.g. , maintaining altitude or heading in turbulence), the 
sampling time must be based on other logical considerations, derived usually 
from experience. For example, the SFTS (Device 2B24) is programed to 
sample performances in ten-second intervale. The question of the relia- 
bility of the sampling intervals must be considered carefully in terms of 
minimum man-system response times and in terms of specific task char- 
acteristics (e.g. , aircraft roll oscillations of three seconds per cycle should 
be sampled for several complete cycles per interval). A capability is also 
desired which enables the instructor to interrogate any single parameter or 
event at any given time in the mission cycle and receive an indication of the 
deviation from the reference value of that parameter or event. 

3.4.7.4.2 Organizing the measurement data— Since the computer is 
utilized for scoring, evaluating and recording performance, a word is in 
order concerning the organization of this information in determining the 
progress and outcomes of training. . m tme histories of performance are 
obtainable as is information on event or status situations due to their binary 
nature. Frequency data are likewise easily assembled and manual control 
scores (e.g* , integrated error rates) can be quickly computed. Larger 
segments of performance and terminal (system) outputs are also achievable 
directly. The computer is able to combine, integrate and weight scores 
and compare the results against defined error tolerance criteria. Thus, 
design decisions must concern the following: 

• Combining, integrating the performance 
dimensions /parameters scored. 

• Weighting of scores and priority ranking of 
scored performance for error and situational 
display at the instructor station. 
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• Providing feedback of performance information 
to the student (error presented immediately as 
accrued , on primary or secondary displays in the 
trainee station) . 

3.4.7.5 Display and Recording of Measurement Information. Computer 
programing is required to 1) monitor the critical aspects o£ student per- 
formance, 2) score all students objectively against standard parameters f 
3) relay pertinent performance and error information to instructor console 
CRT displays, 4) deliver performance cues and performance information 
to display at each trainee station (for example, augmented feedback of per- 
formance information in the form of alphanumeric error indication), and 
5) provide a record of performance for all students throughout each mission. 
The manner in which student performance information is presented at the 
instructor station is a key human factors input to design. 

3.4.7. 5. 1 Computer-Generated Displays — Computer-generated displays 
of student performance information include the following information 
requirements: 

• display of error information and situational 
information (student performance and system 
status). 

• multi-formatting and successive-page formatting 
to provide the necessary performance information 
when required (speed of change and flexibility) for 
the various instructional modes (monitoring and 
control, evaluation, demonstration, reinstruction). 

— • both continuous error indications and the display of 

error in excess of preset tolerances, as well as 
performance and status information on-demand. 

• CRT displays include both alphanumeric and graphic 
page formats with associated selection controls 
(e.g., lightpen, keyboard input devices). 

• error alert indication provides an automatic display 
(and concomitant problem halt) of student error when the 
error exceeds the limits established for a defined 
error class(es) or parameters). 

The CRT display requirements for the measurement of per- 
formance vary considerably among classes of training device so much so 
that the information and formatting details are specific to the device under 
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consideration. Several examples are shown below to provide the reader 
a sampling from a range of possibilities . Figure 25 shows an error analysis 
display for the chaff system in an airborne Electronic Warfare simulator. 
A Graphic Interactive Display (GRID) is shown which consists of an alpha- 
numeric vector CRT display and a lightpen for executing the monitoring and 
control functions. Figure 26 shows a GRID page format for the error alert 
mode for the EW system in the previous figure. Figure 27 shows a display 
of aircraft out-of- tolerance mode proposed for the Air Na vigation Trainer 
(Bark, et al 1969). This shows the preset air lane tolerance band. An air- 
craft appears on this display whenever it deviates from the established flight 
path. The aircraft which is most seriously in error is labeled "most criti- 
cal aircraft" and is identified in a location separate from other errant air- 
craft. A more detailed discussion of data handling and display requirements 
is presented in Chapter 3.4.3. 

3.4.7.5.2 Hard Copy Computer Outputs — Since the digital computer 
is capable of recording error deviations and all time and event happenings, 
a variety of records of student performance may be provided via hard copy 
printouts. This printout can be to the level of switch activations in the exact 
sequence of manipulations or indications that parameter envelops have been 
exceeded. Grosser levels of call-out may also be specified. The record 
of performance serves two purposes. It is used for mission monitoring 
(eriroute) and for post-exercise critique as a means of controlling the 
progress and outcomes of instruction. It is also used for school record- 
keeping and for developing a normative data base of performance informa- 
tion. (The concern for quality control in the production of trained personnel 
is a legitimate issue for design. Measurement data provide the basis for 
determining the quality level of the student population trained in the device.) 
The formatting requirements for retrievable hard copy depend on the infor- 
mation desired and how it must be combined and used. The formats should 
satisfy the following: 

• out-of- tolerance errors — Printouts are provided 
whenever the value of a particular parameter ex- 
ceeds the preset range of allowable parameter 
deviation. This includes, the elapsed time at 
which the parameter(s) deviation occurred and 
when a return to within-tolerance is achieved; 

or the identification of all critical deviations, 
the problem time, response, event or position 
in a maneuver . 

• time and event printouts — These present the total 
performance record, and are used both for critique 
and for school record-keeping functions. An addi- 
tional use is in evaluation (checkride) missions 
where detailed indications are desired on the progress 
and outcome of the mission. 
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• total error printouts --These are complete error 
printouts across mission time and are used for 
critique of training* Error indications are pre- 
sented by classes and trends of errors are identified* 

• summaries of error scoring — These are also used 
for critique purposes, in the instructional sense, 
since they represent short summaries of errors 
made per error class, scores achieved per mission 
segment, total score for the mission, etc. The 
value of this type of single-page summary per mission 
is that a file can be kept simply on each student which 
is useful in diagnosing performance error and for 
trend depiction* 

The information and the formatting requirements for hard copy records of 
performance are discussed in detail in Chapter 3.4.13 dealing with post- 
exercise capabilities . , 

Controls must be provided for instructing the hard copy 
printer to provide particular records on demand. For example: 

• to provide a summary of out-6f- tolerance param- 
eters during the training exercise, such informa- 
tion being keyed to the out- of -tolerance information 
being displayed on the CRT. 

• • to provide a record of the most recent procedural 
task or the one in progress. 

3.4.7.6 Some Additional Considerations. 

3.4.7.6. 1 Conditions Under Which to Measure. Performance- -Once the 
measurement requirements have been determined, the task conditions under 
which performance is measured must be specified. As with the tasks them- 
selves, the conditions under which performance is observed will necessarily 
represent a sampling of the anticipated rea} conditions . In selecting the 
sample of conditions that will prevail, two general rules should be considered: 

• 'Those conditions should be selected which are 
known (or suspected) to have the greater influence 
on performance. 

• Within those conditions selected, the extremes or 
limits anticipated should be included* 
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Task conditions are independent of what measures will be 
used (i.e. , decide on what measurements are desired, then determine the 
conditions). The conditions of measurement, however, do influence when 
the measurements will be taken and in some instances how often the meas- 
ures will be repeated. For example, when load conditions vary in a mission 
sequence, it may be appropriate to take measurements at selected points in 
order to indicate system status or performance at these times. 

Emphasis must be placed on the standardization of test con- 
ditions and environments. Not only should measures be standard (used in 
similar ways with the same meaning by the personnel involved) but also, 
the general environment in which the measurement is to be taken should be 
standard. Adherence to this insures that measures will be strictly com- 
parable from one student or team to ariother and from one time to another. 

Two types of variations in the conditions under which perform- 
ance can be observed are apparent. These are, tack conditions and environ- 
mental conditions. Measurement should be conducted under a variety of 
task loadings representative of those which may be or are encountered in 
operations. Task loading should be systematically manipulated to induce 
stress evoked by situations requiring unusual or complex decisions and 
emergency situations which.give rise to additional task demands. 

It is important also to conduct measurement under the more 
important environmental conditions. These include: acceleration forces, 
movement about axes, sound and vibration, temperature extremes, and the 
range of visual conditions. In order to provide the selected conditions in a 
uniform manner throughout training arid measurement, design should provide 
for. calibration of the simulated environment. ' 

3.4.7.6.2 Quality Control- -The recording of student performance and 
error information for off-line record-keeping analysis (for all students in 
a course) with a program for the analysis of such information, is a desirable 
capability in a training device. For example, programing could be written 
to permit the recording of clasoes of students, performances of students 
assigned to given instructors, performances of students compared to the 
total (cumulative) student norm, etc. With this data base, deficiencies in 
instruction may be efficiently identified, thus facilitating needed revisions. 
Cumulative student data enables the development of a normative data base 
for quality control in instruction. 

An example of automatic quality control of training data is 
provided by the Synthetic Flight Training System (Device 2B24). An auto- 
matic quality control of training-data processing program is specified which 
accepts as inputs, the checkride performance data together with the identi- 
fication of each student, his training group and flight instructor and other 
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descriptive data for comparing the performance of groups* The program 
output data is in hard copy graph form, showing the frequency and cumu- 
lative time of error for various sortings (e.g., by student class, by 
instructor, etc.) for the following: 

• for -each maneuver comprising the checkride 
(e.g., ILS, ADF approach, GCA, instrument 
takeoff, et al). 

• for each parameter identified for scoring without 
respect to maneuver. 

• for each parameter separately by checkride maneuver. 

Each graph contains separate plots of the same data for at least one com- 
parison group (e.g. , the cumulative corresponding data for a preceding 
student group) which may be identified as the reference standard. The pro- 
gram includes an indication of the statistical significance of any deviation 
in performance between the group under examination and the comparison 
gr^up. The program is run off-line. 
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3.4.8 PROCEDURAL PERFORMANCE—DISPLAY AND RECORDING 

A desirable instructional feature is the capability for real-time 
monitoring and evaluation of student procedural (checklist) performance. 
This is another automated performance measurement technique which 
provides an assessment capability for instructor use in developing effec- 
tive training strategies. It is relevant to situations requiring substantial 
amounts of procedures -fo!lowing behavior involving both normal and emer- 
gency sequences. At present, this type of measurement capability is most 
appropriate to flight simulators and serves as a means for a display and 
control of checklist procedural performance (for example, as defined in the 
NATOPS flight manual for the design basis aircraft). 

The displays and controls for achieving this computer monitoring 
of checklist events include the following. 

• CRT display for presenting the procedures to be accomplished 
and student performance data in a single integrated display. 

• keyboard control input to the computer for direct writing on 
the CRT display. The keyboard provides an alphanumeric code 
with the requisite number of displayable characters. 

• programable keyfield to transmit control function codes to the 
computer as selected by the instructor. Coded overlays are 
required for insertion into the keyfield. 

• display generator for accepting computer and keyboard inputs 
and for driving the display. 

• hard copy printout available of the procedural performance 
for debriefing and for school record-keeping. 

3.4.8.1 Design Basis. The information requirements that must be 
satisfied to achieve this capability involve the following classes. 

3.4.8. 1. 1 Modes of Operation — Several modes of operation are re- 
quired to employ this subsystem both enroute during the exercise and off- 
line (pre-* and post-mission phases). 

• problem mode--This is the enroute exercise mode which 
provides for the monitoring of student checklist activities, 
controlling the procedural sequencing (start, stop, reset, 
procedural repeat), and recording student procedural 
performance for subsequent display and debriefing. 
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edit mode — This is an off-line mode which enables the 
adding , deleting and changing the sequence of steps in 
a procedure(s), 

daily readiness check program— This is an off-line mode 
for monitoring and controlling the checkout of the training 
device. 

replay mode— This is an off-line mode for calling .up a 
hard copy of the printout of the stored data for debriefing. 

inspect and change memory locations mode- -This is an 
off-line mode which provides the means for inspecting and 
changing all memory locations in the main trainer com- 
puter and for requesting a hard copy code dump of all 
changes made. 

3.4.8. 1.2 Procedures To Be Trained--The representative list of pro- 
cedures that will be installed for training (both normal and emergency) are 
identified together with the step sequences for each procedure. 

3.4.8. 1.3 Display Presentation-- The following classes of information 
must be accounted for. 

• display format— This includes: the procedure title; the 
sequence of steps for each procedure; the number of 
pages or half-pages for each procedure; the number of 
trials fo* each defined checklist sequence; out- of - 
sequence page number (for the page rot in current view); 
and preview area for alphanumeric keyboard entries. 

• display density — This defines the number of characters 
(alphanumeric) and format that describe the most com- 
plex display (number of lines and number of characters, 
per line). 

• a selective blinking feature serves to indicate errors of 
omission or commission to the instructor. 

• cursor- -This is the means for positioning the inputs from 
the keyboard and for erasing errors. 

• display characteristics — This includes size, shape, 
location, refresh rate and resolution (as an input for 
engineering definition of type of display, e.g. , TV 
monitor, and character generation technique). 
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. controls- This refers to the keyboard requirements for 
instructor inputs, e.g., special keyboards (alphanumeric 
and functions); keyboard coding, number of keys and lay- 
out; keyboard usage per operational mode. 

. overlay design— This is provided for each operational 
mode . 

.3.4.8.2 Design Example: Trainee Monitoring System. An example of 
a design for real-time monitoring and evaluation of trainee procedural per- 
formance is provided next. This is the Trainee Monitoring System (TMS) 
proposed for Device 2F100, simulating the A-7E aircraft (NTDC 1970). 
The TMS provides an additional data source for promoting objective instruc- 
tion and performance evaluation. The procedures -following sequences dur- 
ing ground and inflight operations are computer monitored and student pro- 
cedural performance (errors of omission and commission in procedural 
checks) is displayed at the instructor console. ■ 

A proposed CRT display system consists of a display generator, 
a TV monitor, an alphanumeric keyboard and a programable keyfield (over- 
lays) . The TMS is an interactive display system providing the following 
modes of operation. 

• PROBLEM mode— Enroute monitoring of student 
action and the control of data storage for replay. 

• EDIT mode— Off-line call up and modification of 
procedures, 

• DRED mode- -Off-line monitor and control of the 
daily readiness check program. 

• REPLAY mode— Off-line call up of stored procedures 
for debriefing. 

• INCH mode --Off-line inspection and change of memory 
locations • 

The TMS capability enables the instructor to perform the following: 

• start, stop and reset the training problem. 

• edit operational procedures, adding, changing and 
deleting the specified procedural steps, 

• monitor and evaluate student performance. 
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• record student performance for later replay and 
for debriefing. 

• monitor and control the daily readiness check program. 

• inspect and change memory locations. 

The specifics which follow pertain to the structuring and con- 
trolling of procedural training during an actual exercise (i.e. , operating 
in the PROBLEM mode). An analysis of the procedures specified in the 
NATOPS flight manual for the A-7E aircraft (NAVAIR 01-45-AAE-l) indi- 
cates a total of 31 norma) and emergency procedures pertinent to the 
present training situation.' Table 14 lists these procedures. Of this total, 
the "prestarting cockpit procedures" exceeds all others in the number of 
steps required in the sequence (57 steps) and thus is utilized as a basis for 
organizing the display presentation (i.e. , most complex case). 

3.4.8.2. 1 Display Format — Four independent displays are generated 
to accommodate all operational modes. Each display has the following 
characteristics. 

• total number of characters or symbols - 3456 

• characters or symbols per line - 72 

• number of lines - 48 

• blink- -character selectable 

• white-on-black and black-on-white characters 

• character and symbol set- -95, including graphics 

• cursor 

• refresh rate, 50 fields per second 

Each checklist procedure is considered as a display format. 
Each format is organized into half-page displays with each display consist-' 
ing of 15 steps. The first and second half-pages of a format are displayed 
simultaneously in the upper and lower halves of the CRT. Upon completion 
of the first half- page of a procedure, the second half-page will replace it 
at the top of the display (automatically or manually via keyboard) and a 
third half-page will fill the lower half of the display. This is repeated 
until all steps of a procedure are completed, at which time the next pro- 
cedure in sequence is automatically called up, unless overridden by the 
instructor. 

» 

The format of the display presentation (PROBLEM mode) is 
shown in Figure 28. The normal prestarting cockpit procedure (57 steps) 
provides the example. The displayed information is as follows: 
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TABLE 14, NORMAL AND EMERGENCY PROCEDURES 
FOR A-7E AIRCRAFT (NATOPS FLIGHT 
MANUAL NAVAIR Ol-45-AAE-l) 

1 . Prestarting cockpit procedures -maximum of 57 steps - normal 
procedure* 

2. Engine start ground - maximum of 10 steps - normal procedure. 

3. Pre- takeoff - maximum of 15 steps - normal procedure. 

4. Pre- landing - maximum of 13 steps - normal procedure. 

5. Ejection (primary and secondary) - maximum of 13 steps - 
emergency procedure. 

6. Engine fire ground - maximum 5 steps - emergency procedure. 

7. Engine fire flight - maximum 5 steps - emergency procedure. 

8. Refueling (inflight) - maximum 14 steps - normal procedure. 

9. Fuel control failure - maximum 5 steps - emergency procedure. 

10* Fuel boost pump failure (single pump and pumps 1 and 2) 1 step 
and 3 steps - emergency procedure. 

11. Wing fuel transfer - 2 steps - normal procedure. 

12. Wing fuel transfer failure - maximum 3 steps - emergency procedure. 

13. Master generator failure - maximum 4 steps - emergency procedure. 
14# Pitch trim failure - maximum 6 steps - emergency procedure. 

15. Roll trim failure - maximum 3 steps - emergency procedure. 

16. PCI System failure - maximum 8 steps - emergency procedure. 

17. t PC2 System failure - maximum 8 steps - emergency procedure. 

18. PC3 System failure - maximum 8 steps - emergency procedure. 

19. PCI and PC2 failure - maximim 9 steps - emergency procedure. 
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TABLE 14. NORMAL AND EMERGENCY PROCEDURES FOR 
A-7E AIRCRAFT (NATOPS FLIGHT MANUAL 
NAVAIR 01-45-AAE-l)(Continued) 

20. PCI and PC 3 failure - maximum 11 steps - emergency procedure, 

21* PC2 and PC3 failure - maximum 11 steps - emergency procedure. 

22. Canopy jettison - maximum 4 steps emergency procedure. 

23. Instrument Check list - maximum 17 steps - normal procedure. 

24. After landing - maximim 8 steps - normal procedure. 

25. Before engine shutdown - maximum 7 steps - normal procedure. 
26* Engine shutdown - maximum 3 steps - normal procedure* 

27* Before. leaving aircraft - maximum 3 steps - normal procedure. 

28* Taxi - maximum 9 steps - normal procedure. 

29. Before takeoff - maximum 16 steps - normal procedure. 

30. Flameout - maximum 19 steps - emergency procedure. 

31. Speed brake failure to retract - maximum 4 steps - emergency 
procedure. 
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• procedure title. 

• all procedural steps in sequence of accomplishment. 

• procedure half-page number (2 half-pages per display) 
, out-of-sequence half-page number for out-of-sequence 

step performed on the half -page not in view. 

• preview area for alphanumeric keyboard entries. 

• numerical character blinking when out-of-sequence 
step occurs 

• three sets of "sequence performed" step numbers (in- 
structor has the capability to repeat a procedure a 
maximum of 3 times) 

3,4.8.2.2 Controls --Instructor call up of a particular procedure is 
initiated through a function keyboard containing 32 keys. These keys to- 
gether with- coded overlays are used to control the TMS in PROBLEM, 
EDIT, REPLAY, INCH and DRED modes. A capability for up to 16 
overlays is provided. Inserting an overlay will automatically select the 
pertinent operational mode. For the PROBLEM mode, a function keyboard 
overlay provides the following functions: 

• procedures (a maximum of thirty-five requiring multiple 
overlays) 

• procedure advance (manual or automatic) 

• half- page advance 

• start 

• reset 

• store 

• replay 

• blinking control 

An alphanumeric keyboard provides a serial binary code for 
decoding and direct writing on the CRT TV monitor. The standard alpha- 
numeric repertoire of 26 alphabetical and 10 numerics plus special symbols 
for identifying or coding procedural steps is provided. Control keys for 
cursor deployment are also included. 
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3.4.9 CAPABILITY AND CONTROLS FOR INPUTS TO THE COMPUTER 

Means are required for inserting instructions to the computer to 
achieve the desired CRT display outputs relative to the structure and control 
of training. The criteria for determining the design of manual input devices 
are based on the compatibility of the candidate choices with the software pro- 
graming. Specifically, this concerns the compatibility with a variety of 
tabular and graphic display formats, flexibility of usage and speed of change 
in information display, efficiency of panel space usage, and ease of training 
the instructor in its usage. 

Our discussion centers on keyboard design since this at present is 
the conventional device selection for the manual input capability, although 
the use of the lightpen is a desirable design option. * Various purposes are 
served by this inputting means, both off-line and on-line including scenario 
or problem generation/modification, enroute problem control, student per- 
formance record editing, and trainer maintenance. Two basic kinds of 
keyboard serve the various data entry purposes. 

• alphanumeric keyboards serve as the primary means of 
communication between the instructor and the computer 
relative to the structure, Control and modification of the 
training and the simulation-conditions. 

• function keyboards serve in the selection of defined trainer 
. operating modes (primary trainer control means) during 

the course of an actual training exercise. The function 
keyboard is also utilized as the primary means of control for the 
trainee monitor system (PROBLEM, EDIT, DRED, INCH 
and REPLAY modes— see Chapter 3.4.8). 

The information requirements that must be satisfied in the design 
of manual input devices for accessing the computer include the following: 

• the general and specific functions (on-line and off-line 
to be served by the input device(s). 

• the display modes associated with the input device(s). 

• keyboard(s) size and placement on the instructor console. 



Graphic interactive displays consisting of an integrated-circuit controller, 
an alphanumeric vector CRT Bisplay and a lightpen have been designed for 
executing the monitor and control functions at the instructor console. This 
is an effective means for displaying student performance data and mission 
information to the instructor and for providing the required control for 
instructional actions. 
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. alphanumeric keyboard design — Number of keys (alpha- 
numeric and functions), key layout (e.g. , standard 
typewriter layout), key size and color coding. 

# function keyboard design--Definition of functions, coded 
overlay capability for all problem modes. 

# electronic interlock requirements (error correction, 
simultaneous depression of two keys, keyboard lockout 
of a channel during computer I/O with that channel). 

# speed of operation. 

# definition of operational sequences. 

3.4.9. 1 Design Example. An example of manual input device require- 
ments is the keyboard design proposed for an operational flight trainer 
(Device 2F100, Goodyear 1971). An alphanumeric and a functions keyboard 
are specified. The alphanumeric keyboard is equipped with 48 alphanumeric 
keys and 21 function keys. A standard typewriter layout is employed and 
the necessary electronic interlocks are provided to prevent garbling of 
codes when two or more keys are depressed simultaneously and to lock 
out the keyboard from a channel during computer I/O with that channel. 
The 21 function keys serve the following: insert, space bar, erase page, 
carriage return and line feed, word erase, cursor positioning (4 positions), 
color code (4 optional codes), blink, home, erase-line, backspace and line 
feed, clear, shift, shift lock and repeat. The alphanumeric keyboard lay- 
out is shown in Figure 29 . 

The performance specifications for the training device also call 
for a thirty-two key function keyboard, with special coded overlays for the 
PROBLEM, EDIT, DRED, INCH and REPLAY modes. An example of the 
overlay for the PROBLEM mode is shown in Figure 30. To enable the in- 
structor to access quickly all pertinent data and make problem changes 
according to student actions, two overlays are provided (i.e. , to accom- 
modate two sets of procedures). Each overlay has eight common functions 
and four unused keys. The keyboard design reflects the following: 

" # The less a key is used, the nearer to the back 
and right of the keyboard it is placed. 

# If a function is used on more than one overlay, 
the same key is used. 

# Associated functions are placed in rows. 
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Figure 29. Alphanumeric keyboard. 
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Figure 30. Overlay, PROBLEM mode. 
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The sequence of events begins at the bottom row 
of keys and works up. . 

No more than two keys will be used to access, a 
function. 

Lights are incorporated in the keys to indicate the 
function being processed. 

Inserting the overlays will automatically select 
PROBLEM, EDIT, REPLAY, DRED and INCH 
modes. 
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3.4.10 SOFTWARE CAPABILITY FOR DEVELOPING AUTOMATED 
ADAPTIVE STRATEGIES 

The emphasis on machine or computer control in the development 
of the training capability places severe requirements on the software pro- 
graming. Computer programing is the means for formalizing the decision 
logic tailored to student performance based on a responsive monitoring, 
evaluation and scoring capability. 

Automatic adaptive sequencing as a technique for programing the 
computer is a new approach for training device design* and not much data 
are available for design use. Adaptive simulation is a difficult undertaking 
from the standpoint of instructional validity, and at this time, many prob- 
lems and issues must be resolved before firm guidelines can be established 
for implementing this technique. Although our purpose in this report is not 
to stress the inadequacies in design data , it must be made clear that inher- 
ent difficulties exist .in implementing an automatic adaptive system that will 
satisfy the instructional expectations. This prologue is thus warranted to 
place the material which follows in perspective. 

3.4. 10. 1 Adaptive Simulation as a Design Option. Perhaps the signifi- 
cant question for design concerns the efficacy of adaptive simulation for 
training. Although a number of interesting relationships have been demon- 
strated in the laboratory, no substantial assessments are available which 
evaluate the effectiveness of adaptive simulation as a practical training 
method for producing the skills required in the operational environment. 
However, it is not difficult to favor adaptive training as a concept since 
logically all training is (ideally) adaptive. Hence, many theoretical advan- 
tages inhere in adaptive simulation. As a technique for presenting infor- 
mation to the student it has made explicit a number of features that are 
intuitively desirable. These include the following: 

• Formal structure and control of training requiring 

a deliberate logic (tailoring learning to the individual 
based on rational decisions about performance). 

• Precise control of task loading. 

. . Objective adjustment of the difficulty level of tasks. 

• Structuring the link between the student's performance 
and the presented task. 

• Emphasis on measurement during the training process. 
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• Requirement to analyze all the variables that can 
influence student performance and task difficulty. 

• Record of the training process. 

While these features are not peculiar to adaptive simulation, 
all are pertinent to the technique. 

Assuming the efficacy of adaptive simulation, what training 
problems are adaptive techniques best suited to solve? This is a relevant ' 
question for design, since it appears that not all training will benefit from 
this approach. To date, it has been applied almost totally to tracking be- 
havior and perceptual motor skills, so much so that it has become a major 
preoccupation of flight simulation. Some insights are provided below (see 
also McGrath and Harris 1971). 

• Adaptive training methods are most probably called 
for in perceptual-motor tasks that are too difficult 
for a student to handle at the outset. 

• Adaptive training is called for when the computer can 
substitute for the instructor and when the task is of 
such a high order of difficulty that it cannot be mastered 
unless it is broken down into component parts. 

• Adaptive training methods seem warranted when training 
is already adaptive, but instructor controlled, and when 
it is desirable for reasons of cost, standardization, or 
(when it can be demonstrated) training effectiveness to 
mechanize the instructor's adaptive function. These 
methods are also appropriate when it can be shown 

that training time is significantly reduced or that a 
higher level of skill is reached in the same amount 
of time by replacing a fixed with an adaptive training 
schedule. The criterion should be cost-effectiveness, 
broadly interpreted and applied. 

• Adaptive training systems are probably most useful 
in situations requiring considerable overlearning and 
high retention over time. They are most appropriate 
in systems where variables can be closely, perhaps 
infinitely, controlled; but application to part- task/ 
whole -task grouped learning patterns may not be 
amenable to the close control over levels of difficulty 
that seems essential to the technique. The important 
problem is the relevance of the adaptive variable used 
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as a forcing function (turbulence, for example) to 
the task for which the student is being trained. 

• Automated adaptive training may be used for the 
purposes of reducing the need for human instructors, 
standardizing training procedures, making the deci- 
sion function more objective and rational, or achiev- 
ing greater precision of control in presenting tasks 
and instructional materials. These are important 
goals , but they can be met by oth?sr methods of 
automated, structured, or programed training that 
are not necessarily adaptive. The purposes that 
can be served uniquely by adaptive training must be 
those which are served by maintaining a continuously 
closed loop between the student's performance output 
and the task-stimuli input. 

Thus, adaptive training methods are worthy of consideration when: 1) the 
task to be learned is of sufficient difficulty to require a significant amount 
of time to achieve mastery, 2) task improvement is a linear or progressive 
phenomenon, 3) the training situation lends itself to automated controls, 

4) the subject population can be controlled to the extent necessary, and 

5) the training task bears sufficient resemblance to the final behavioral 
model to have a reasonable transfer value. 

Adaptive techniques automate an important instructor function, 
that of problem selection (that is, progressing through a graduated series 
of exercises to achieve the optimum training strategy). This involves the 
automation of two other functions: scoring of performance so that adapta- 
tion can occur; and providing information so that the student will know where 
to direct effort at reducing his error rate. In defining the information re- 
quirements for design, the following elements must be resolved. 

1 . Selecting the relevant adaptive variables (task 
modification to systematically change its difficulty). 

2. Defining the performance measurement requirements 
(continuous/repetitive scoring of performance). 

3. Specifying the adaptive logic (the function used to auto- 
matically adjust task difficulty based upon measured 
performance). 

4. Determining the performance information to display. 

3.4. 10.2 Selecting Adaptive Variables. An initial decision is the choice 
of the parameter or variable which will be manipulated to adjust task 
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difficulty. The task must be scalable along some dimension of graduated 
difficulty, either continuously or in step fashion. The adaptive variable 
selected must systematically affect the difficulty of the task. 

A number of criteria for the selection of adaptive variables are 
summarized below to assist the human factors specialist in this important 
determination. The adaptive variable: 

9 must be clearly relevant to the operational task 
being trained; its variation should be along dimen- 
sions relevant to the skill being taught* 

• must be capable of being described and quantified, 
i.e. , it must be capable of being measured. 

• must be related to task difficulty; changes in the 
adaptive variable must bring about changes in 
level of performance* 

• must be capable of being varied in a systematic 
way; although this seems obvious, in a practical 
situation, it may not be easy to do. 

• When used in devices for training for transfer to 
a different system, it should not inhibit or inter- 
fere with transfer of training* 

• must be capable of being adjusted over a, range 
commensurate with the student's skill both in 
increasing and decreasing difficulty. For example, 
the use of turbulence as an adaptive variable might 
be limited in its application because in some systems 
(e.g. , the helicopter), the task difficulty for the 
beginning student must be reduced to some level 
below that of normal operation in calm air* That 

is, for the beginner, the task may need to be made 
easier than it actually is in the real system for most 
efficient training. 

• must do no obvious violence to the training situation, 
i.e. , it should be realistic in the sense that it varies 
in ways in which the real task varies so that it is 
acceptable to the student; the adaptive variable must 
not change the basic task so that it becomes strange and 
unreal to the student and not acceptable to him as a 
training situation* 
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3.4. 10*3 Classes of Adaptive Variables. Various classes of adaptive 
variables are relevant to training system design. These are described 
next. The presentation tacitly assumes the rejoinder that the surety of 
selection awaits more empirical evidence. Within each of the classes of 
adaptive variable there is a variety of simulator features that can be varied, 
so that these singly or in combination provide a substantial range of possi- 
bilities. It is well to consider only a single continuum of difficulty for design. 
At present, the interactions among multiple dimensions are not well under- 
stood, hence complex scoring schemes are not encouraged without further 
research. 

3.4. 10. 3. 1 External Forcing Function — Turbulence has been most in- 
vestigated as an adaptive variable in flight simulation, employed to make 
the control task more difficult. Obviously, its use at the outset of training 
is limited. Abo, in systems difficult to control because of systems dyna- 
mics, difficulty level cannot be decreased below the level set by the system 
dynamics, hence, turbulence may not be a prime choice. In defining the 
turbulence spectrum, decisions involve variations in frequency, amplitude 
or both. Bandwidth of the forcing function appears critical in that it may 
interact with rate of change in difficulty and the length of the interval during 
which performance is measured. Wind velocity (head wind or tail wind) 
appears useful in that increases or decreases in ground speed influence 
difficulty level in terms of time constraints. Crosswinds have the effect 
of increasing the workload for the student in ground tracking since aircraft 
headings must consider this wind vector. 

3.4. 10. 3.2 Vehicle Characteristics—Several types of difficulty factors 
have been identified as promising for manipulating task complexity. These 
include: 

• control 'damping- -increasing the moment of 
inertia on the control motions can be used to 
make the task less difficult. With damping, 
the student is not required to respond as 
quickly as without damping, hence loss of 
aircraft control, especially during initial 
training, will be less likely. 

• center of gravity changes--CG shifting re- 
sulting from fuel loads may vary the student's 
workload in aircraft control. 

• malfunctions --relevant system malfunctions 
are pertinent to graduated task difficulty levels. 
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3.4. 10. 3. 3 Signal Characteristics--This includes degradation in signal 
characteristics (e.g. , noise); the incidence of signals per. unit of time in a 
mission context (e.g. , density of ECM signals of varying priority classes 
for a given time unit), and simultaneity of signals at a given time (e.g. , two 
or more high priority EW signals as immediate threats in the airborne 
environment) . 

3.4. 10.3.4 Secondary Task Loading — The secondary task has been sug- 
gested as an adaptive variable , i.e. , adjusting the primary task difficulty 
by secondary task loading. However , the use of secondary tasks must be 
considered carefully in training device design , if only from the viewpoint of 
task changing or realism. Of concern is the issue of whether secondary 
tasks change task difficulty or actually change the task. For flight training, 
the variables of communications and command pacing are perceived as 
realistic , but they may add to task structure more than just increasing the 
task difficulty. In other words, the object is not to just increase difficulty 
but also to make the task more realistic. If the secondary task is to be 
considered as an adaptive variable , then attention must be paid to, how the 
student perceives the two tasks, the relative difficulty level of the two tasks, 
the amount of student activity involved in performing both tasks and the 
manipulation rates for the secondary task. It seems best to argue that the 
secondary task technique is more useful as a laboratory research tool rather 
than as an adaptive variable for operational training. 

3.4. 10. 3. 5 System Quickening- -In system quickening the actual output 
of the system is changed by the quickening. This is in distinction to display 
quickening wherein error scores are derived for displayed error rather 
than for system output* The investigations thus far, however , suggest that 
quickening and aiding are of relatively low priority as candidates for adap- 
tive variables. 

3.4. 10*3.6 Effective Time Constants-Change in the effective time con- 
stant of the vehicle response (t e ), has been proposed as an adaptive variable. 
The effective time constant of the man-machine system is a measure of the 
time between control input initiation and the operator's detection of the vehi- 
cle response to the control input, i.e. , the speed with which the operator 
detects the results of a control movement. It is a measure of the respon- 
siveness of the vehicle to control movements and is made up of the speed 
of the vehicle response and the operator^ threshold for detecting changes 
resulting from control inputs. Matheny and Norman(1968) showed t^ to be 
related to rate of learning and to final level of precision of control. Level 
of performance was related to the interactive effects of t e and system gain, 
with the effects of gain most prominent during initial learning and t e more 
important in determining final level of performance. For an adaptive train- 
ing sequence, optimum gain and t e would be provided in initial training. 
As training progressed, gain and t e would be adjusted to achieve the task 
difficulty progression. 
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3.4. 10.3.7 Changes in Control Order— Hudson (1964) suggested that 
changes in control order during training is an important issue;" optimum 
learning may occur when the practice task involves, moderate or average 
level of difficulty regardless of the level of difficulty of the criterion task. 
Level of training achieved is largely a function of level of difficulty durinc 
practice. 6 

3.4. 10.4 Measuring Performance. Obtaining relevant and reliable per- 
formance measures is as critical as is the selection of the adaptive vari- 
ables for training, so much so that the success of adaptive training is 
predicated on adequate measurement. The research to date has focused 
on tracking tasks using traditional performance indices (absolute error, 
time -on- tar get) as performance criteria to adjust an adaptive variable 
affecting the task. Experience to date with adaptive measurement has been 
sparse and has centered primarily on flight simulation. For example, in 
the Synthetic Flight Training System, Device 2B24, measurement considers 
the important parameters in aircraft control and performance tolerances 
have been set based on best estimates guided by experience. The specified 
performance parameters and the suggested performance tolerances have 
already been shown in Table 13. 

A research study (Kelley and Wargo 1968) recommends the fol- 
lowing. Possible performance measurements for the training device should 
begin with basic vehicle control parameters— airspeed, altitude, and attitude 
during steady state flight. Error tolerances for each parameter must be 
established empirically for various flight conditions. Measurement should 
also consider task components such as vehicle status monitoring, navigation 
and communication even though these are more difficult to measure effec- 
tively. The scoring of flight path control during maneuvers can be achieved 
by computing repetitively the response of an "ideal" pilot which is compared 
to the student response on the basis of vehicle output parameters since there 
is a wide range of operator control manipulations that will result in the de- 
sired vehicle response. Instantaneous performance should be measured in 
terms of basic vehicle state parameters. The parameters suggested are: 
three coordinates of position; airspeed; heading; rate of climb/ descent/ and 
pitch (roll and cross -track velocity could be used if found to be important). 
A nominal value should be computed for each scoring parameter via a com- 
puter-based model representing the desired vehicle state at each instant, 
and recomputed periodically based on new real-time parameters. Error 
tolerances about the minimal values for each scoring parameter should be 
established (e.g. , four categories ranging from "within tolerance" to "far 
out-of-tolerance"). An accumulating error score should be kept for each 
scoring parameter. Scores not in tolerance should be weighted (incremental). 
Performance on all parameters together should be assessed by logical rules 
(adaptive logic) to determine if adjustments in task difficulty are warranted. 
(In determining the adaptive logic, tolerance limit settings and logic rul .s 
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require examination of score distributions and intercorrelations on tasks 
performed by skilled pilots). 

Another recent research concerned with measurement in the 
flight context (Vreuls and Obermayer 1971) recommends the measurement 
sets shown in Table 15 for the adaptive variables of turbulence and command 
pacing ♦ 

3.4. 10 . 5 Adaptive Logic. A training system is adaptive to the extent that 
task difficulty level is an automatic function of measured performance (no 
instructor intervention). The adaptive change occurs as a result of the 
difference between a preset performance standard and the latest measure- 
ment of performance. Thus, the adaptive logic links the adaptive variables 
and performance measurements. 

The error standard must be established. The student's actual 
moment-to-moment performance is compared against this criterion and 
task difficulty level is adjusted as a result of this comparison. Obviously, 
there is an optimum standard; less than this will degrade training efficiency. 
When the standard is too tight, the error tolerance is small and the adaptive 
system will automatically remain less difficult to the student; when the stand- 
ard is too loose, the student will progress more rapidly to a more difficult 
task configuration. 

The length of the performance sampling periods must be estab- 
lished. When the sampling period is too short, exaggerated changes in 
difficulty level may occur and task difficulty may exceed the student's 
momentary skill level. The design decision must consider the implications 
of slow varying vs. quick varying adaptive tasks, particularly, if large fluc- 
tuations in student performance are expected, since these may force the 
system to adjust difficulty level excessively. In this sense, slow varying 
adaptation represents a more reasonable selection than does quick-varying 
adaptation. 

3. 4. 10. 6 - Display of Performance Information. Supplementary knowledge 
of results should be displayed, even though artificial and not directly related 
to the criterion task. Some digital readout of the momentary status of the 
adaptive variables seems sufficient. Device 2B24, for example, will employ 
a meter which gives the level of difficulty of the system at each moment. A 
two-part score is proposed. The first part (2-3 digits) identifies the mission 
or the problem installed; the second' part (1-2 digits) identifies the level of 
problem difficulty as determined by the intensity of the adaptive variable. 

Indicator lights may be used to depict out-of-tolerance condi- 
tions as they emerge. Augmented feedback may also be provided directly 
on a primary display, for example, in the form of an alphanumeric message 
alerting the student to a specific out-of-tolerance error. 



-222- 



NAVTRADEVCEN 71-C-0013-1 



TABLE 15. MEASUREMENT CANDIDATES FOR 

TURBULENCE AND PACING ADAPTIVE 
VARIABLES 



MEASURE 


FUNCTION OF 


TOLERANCE 


Altitude Error 


Altitude Command 


±100 Ft. 


Altitude Rate 
\ iri ability 


Steady-State Climb/ 
Dive or Hold Command 


±600 Ft. /Minute 


True Airspeed or 
IAS or Mach 


Steady-State Airspeed 
Command ' 


±20 knots or mach 


Roll Average 
Error 1 


Bank Command 


±5° 


g-Error 1 9 2 


Bank Command 


±•2 g for Pacing, 
±•4 g for Turbulence 


Turn Rate Error 1 


Bank Command 


±•2 deg./sec. for 
Pacing 



These measures to be taken from the time a command is first achieved. 



^g-error is the difference between the obtained value and the- desired value, 
(from Vreuls and Obermayer, 1971) 



The issue of whether knowledge of performance information 
should be displayed continuously, periodically or event oriented, and direct- 
ly or indirectly cannot be resolved at this time. It is, nevertheless, of 
importance to design. 

Additional adaptive features of Device 2B24 are worth noting 
since this trainer is the first to include adaptive simulation in design. At 
present, four adaptive variables have been selected for implementation. 
These are: 

• turbulence (either longitudinal axis or roll axis 
emphasis). 

• control damping (increasing the moment of inertia 
on the control motions--i.e. , airframe response) 



;rjc 
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• horizontal wind (headwind or tailwind to modify 
ground speed) 

# cross wind 

Error scoring for the purpose of adjusting task difficulty will be in percent 
time out-of -tolerance for each scored parameter. As it now stands, the 
scores will be treated separately for each parameter rather than being 
combined into a single score. To increase task difficulty, performance 
must be within tolerance for a designated percent of the sample time on all 
parameters being used in a training exercise* The entire task interval has 
been arbitrarily divided into 10-second intervals, i.e., every 10 seconds, 
time out-of rtolerance and error tfate is examined on each parameter. In 
the initial programing, error rate may be set ^t 10 percent. Task difficulty 
will not change if performance is within tolerance 90% of the time during 
the sample period. If performance is acceptable less than 90% of the time, 
task difficulty will decrease; task difficulty will increase if acceptable per- 
formance exceeds these limits. The criterion function is reset to zero 
whenever the difficulty level changes. The magnitude of the adjustment in 
task difficulty will be a function of the particular adaptive variable being 
employed in each problem. 

Each student will receive knowledge of results of progress via 
a two-part digital readout. The first part indicates the problem being 
practiced (this would indicate level of difficulty or stage of training if ex- 
ercises are graduated in difficulty); the second part indicates the level of 
problem difficulty as determined by the intensity of the adaptive variable. 
Thus, a score of 18.50 indicates that problem 18 is in effect with a mod- 
erate intensity level for the adpative variable being employed (i.e. , entry 
difficulty level of 3; exit criterion difficulty level of 9). 

It is emphasized that the above describes initial programing 
requirements since the device is not expected to be on-line prior to 1971. 
Thus, adjustments will be made and concepts may be revised through a 
series of successive approximations until the optimum values are achieved 
as research information and experience with the on-line device accumulates. 

The problems besetting Device 2B24 in implementing adaptive 
sequencing have been considered at an adaptive training workshop at the 
University of Illinois (McGrath and Harris 1971). Excerpts from this 
deliberation are described below. * 

9 The major problem is in the performance measures. 
The validity and the reliability of the proposed meas- 
ures can be questioned. Research is recommended 
on the performance measurement problem before 
very much reliance is placed on the adaptive training 
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feature of the system. For example, percent of time out- 
of-tolerance may be a useful measure for some of the train- 
ing but its usefulness is a function of the validity of the 
tolerances employed. 

9 The 10-second interval of performance measurement 
for determining adaptive changes will probably be too 
short. The available empirical evidence on adaptive 
training seems to favor the "slowly adapting" system, 
and so does theory. Moreover, on the grounds of psycho- 
physics alone, there are few aspects of human performance 
that can be measured reliably in 10 seconds; and if the per- 
formance measurement is unreliable, then, of course, 
the adaptive "logic 11 is reduced to a largely random process. 

9 The time out- of- tolerance measures of performance 
may have to be better defined. It is possible that much 
information of potential value may be lost during the 
"in-tolerance" performance. 

• The planned approach that calls for the student to practice 
control on one axis, then on two axes, and so on, will lead 
to difficulties in trans fering from one task to another in the 
sequence. The freezing of any axis or set of axes may cause 
the student to adapt to response modes that are incompatible 
with response modes required when the axis or axes are 
unfrozen. This situation could result in significant nega- 
tive, or at least neutral, transfer from one task to the 
next. 

. A good start has been made on the choice of adaptive 
variables. Of course, the adaptive logic requires em- 
pirical study with a great deal of attention given to the 
standards employed. The overriding problem, however, 
is that of performance measurement. There is doubt that 
the measurement approach presently planned is reliable; 
or that it will result in effective adaptive task control. 

. A great deal of refinement is going to have to take place 
in the exit criteria and in the way a student is tracked 
through his training. It is anticipated that the 1 to 9 scheme 
for scaling difficulty will have to be revised and that a 
system for accumulating milestone "misses 11 will have to 
be incorporated. Also, student motivation will probably 
not be handled very well by the current scheme. 
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3.4. 1 1 AUTOMATED VERBAL MESSAGES 

In training systems possessing an automated monitoring, evalua- 
tion and scoring capability, there are instructional sequences that benefit 
from the use of prerecorded verbal messages. With automated training 
evaluation, these messages make available coaching information (cues and 
prompts which provide supplementary information about performance before 
or during student response) and error alerting information (augmented 
feedback which provides performance information immediately as accrued, 
i.e. , information feedback during or immediately after student perform- 
ance). They serve also to relieve the instructor from time consuming 
performance monitoring and communications activities, the burden of 
which is significant in multi- student trainers wherein each student performs 
independently. 

At .present, this instructional technique is most appropriate to the 
task requirements in flight simulation. Voice messages are transmitted 
to the cockpit from a computer -driven recorded voice system. Individual 
messages are assembled word by word and formatted into messages under 
computer direction as a consequence of student control or vehicle actions. 
These precise information messages are most appropriate in 1) task situa- 
tions where communications requirements are heavy, e.g., the coaching 
requirements in a Ground Controlled Approach (GCA); and 2) situations 
where the messages are clearly correlated with emergencies or develop- 
ing out -of- tolerance system states, e.g. , performance alerts which 
announce that a measured parameter soon will, or currently exceeds a 
preset error tolerance. 

The requirements for the design of a computer controlled voice 
system include the following categories of information. 

• selection of the instructional situations employing automated 
verbal messages, e.g. , performance alerts (out-of- tolerance), 
emergency procedures, task segments. 

. precise definition of the task structure benefitting from the 
automated voice generating technique. 

9 definition of the word universe appropriate to the communica- 
tion requirements of the task situation. An example is pro- 
vided in Table 16 of a word vocabulary currently utilized by 
NTDC. 

* definition of the number of messages required and the 
structure of each message. 
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TABLE 16: NTDC WORD LIST 



ABOVE 


21 


EXECUTE 


2B 


MINUTE 


35 


SILENCE 


OA 


ACKNOWLEDGE 


61 


FAST 


IF 


MISSED 


75 


(SILENCE) 


00 


AFTER 


Al 


FEET 


6B 


NAUTICAL 


B5 


(SILENCE) 


40 


AGAIN 


13 


FINAL 


AB 


NAVY 


36 


(SILENCE) 


80 


ALPHA 


4A 


FIVE 


07 


NINE 


48 


SIX 


47 


ALSO 


22 


FOR 


2C 


NINER 


88 


SLIGHTLY 


3E 


ALTITUDE 


62 


FORWARD 


56 


NO 


76 


SLOW 


9E 


AND 


A2 


FOUR 


86 


NORMAL 


B6 


SOUTH 


IB 


ANSWER 


53 


FOXTROT 


OC 


NORTH 


98 


SPEED 


5B 


APPROACH 


23 


FROM 


6C 


NOSE 


19 


START 


9B 


APPROACHING 


63 


FURTHER 


AC 


NOT 


37 


STICK 


1C 


ARE 


93 


GLIDE 


2D 


NOVEMBER 


8E 


SURFACE 


7E. 


AT 


A3 


GOLF 


4C 


OBSERVER 


77 


TAKE 


BE . 


AVAILABLE 


24 


GOOD 


6D 


OF 


B7 


TANGO 


90 


BACK 


14 


GUSTY 


AD 


OFF 


59 


THAT'S 


5C 


BE 


96 


HALF 


49 


ON 


38 


THE 


3F 


BEEN 


64 


HEADING 


2E 


ONE 


85 


THESE 


9C 


BEGIN 


A4 


HEAR 


6E 


OR 


78 


THIS 


7F 


BELOW 


25 


HELLO 


17 


OSCAR 


OF 


THOUSAND 


BF 


BRAVO 


8A 


HOTEL 


8C 


OVER 


B8 


THREE 


46 


CANCELLED 


66 


HOW 


AE 


PAPA 


4F 


THRESHOLD 


20 


CHARLIE 


OB 


HUNDRED 


2F 


PATH 


39 


TO 


60 


CLEAR 


A5 


IF 


6F 


PATTERN 


79 


TOUCHDOWN 


AO 


CLEARENCE 


66 


IMMEDIATELY 


57 


PEDDLE 


99 


TOWER 


01 


• CLEARED 


26 


IN 


AF 


PER 


B9 


TRANSMISSION 


41 


CLIMB 


A6 


INCREASE 


97 


PERFORM 


3A 


TURN 


81 


COMPLETE 


27 


INDIA 


OD 


PLEASE 


9F 


TWO 


06 


CONFIRMED 


67 


INFORMATION 


30 


PLUS 


1A 


UNABLE 


02 


CONTACT 


A7 


IS 


70 


POINT 


7A 


UNIFORM 


11 . 


CONROL 


5E 


JULIET 


4D 


PRECISION 


BA 


UP 


ID 


CON1 ROLLER 


28 


KILO 


8D 


PRESENT 


5A 


VK* 1 UK 


41 
7i 


CORRECT 


54 


KNOTS 


B0 


QUARTER 


09 


VISIBILITY 


42 


CORRECTED 


68 


LAND 


31 


QUARTERS 


89 


VISUAL 


82 


CORRECTION 


A8 


LANDING 


71 


QUEBEC 


8F 


VISUALLY 


03 


COURSE 


29 


LEFT 


Bl 


RATE. 


3B 


WELL 


43 


DAY 


69 


LEFT OF 


32 


RECEIVED 


7B 


WEST 


5D 


DECREASE 


94 


LEVEL 


18 


RECOMMEND 5F 


WHEELS 


83 


DEGREES • 


A9 


LIGHTS 


72 


RIGHT 


3B 


WHISKEY 


91 


DELTA 


4B 


LIMA 


0E 


RIGHT OF 


3C 


WIND 


04 


DESCEND 


15 


LOUD 


B2 


ROGER 


7C 


WING 


9D 


DESCENT 


2A 


MAINTAIN 


33 


ROMEO 


10 


WITH 


IE 


DIVE 


S5 


ME 


73 


RUNWAY 


BC 


WITHIN 


44 


DO 


6A 


MIKE 


4E 


SECONDS 


3D 


XRAY 


12 


DOWN 


AA 


MILE 


B3 


SETTINGS 


9A 


YANKEE 


52 


EASE 


05 


MILES 


34 


SEVEN 


87 


YOU 


84 


EAST 


16 


MINIMA 


74 


SHOULD 


7D 


YOUR 


05 


ECHO 


8B 


MINIMUM 


B4 


SIERRA 


50 


ZtfRO 


45 


EIGHT 


08 


MINUS 


58 


SIGrfT 


BD 


ZULU 


92 
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. assignment of message priorities when two or more 
messages are appropriate simultaneously, e.g. , when 
two parameters are out-of-tolerance at any given time. 

. selection of the timing sequence for successive messages 
for the same system state, e.g. , realerting the student at 
given intervals for a parameter that continues to be out- 
of-tolerance. Included here is the decision on automatic 
(computer) halt of the exercise with continued student 
violation of an error envelop. 

. means for modifying the verbal messages and for over- 
riding a message during an exercise or portion thereof. 

Two examples of currently employed voice generation systems 
are provided below. The first is a system which was incorporated in a 
research study which attempted to demonstrate the technical feasibility 
of automating portions of a weapon system trainer (Charles and Johnson 
1971). A ground Controlled Approach (GCA) and emergency procedures 
tasks (F-4 aircraft characteristics) were selected for automated imple- 
mentation on the NTDC Training Device Computer System (TRADEC 
system) and tested with operational pilots. The computer-controlled 
voice system employed the vocabulary list shown in Table 16; messages 
for the GCA and emergency procedure exercises were assembled by the 
Meta-Symbol procedure in this program. The following messages were 
employed in the automated programs . 



CLIMB AND MAINTAIN 2500 FEET, HEADING 045 DEGREES 
CLEARED RUNWAY 04 

CORRECT HEADING IS 

HEADING IS GOOD 

TURN LEFT HEADING 

TURN RIGHT HEADING 

NAVY 123, CONTACT 9 MILES FROM TOUCHDOWN, FINAL 
CONTROLLER 
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NAVY 123, IF ON FINAL AND NO TRANSMISSION IS RECEIVED 
FOR 5 SECONDS TAKE OVER VISUALLY: IF UNABLE EXECUTE 
MISSED APPROACH 

NAVY 123, PRECISION MINIMA 200 FEET, 1/2 MILE; IF RUN- 
WAY NOT IN SIGHT AT MINIMA, COMPLETE APPROACH TO 
TOUCHDOWN IF THE TRANSMISSION TO EXECUTE MISSED 
APPROACH IS NOT RECEIVED 

NAVY 123, ON FINAL, DO NOT ACKNOWLEDGE FURTHER 
TRANSMISSION; APPROACHING GLIDE PATH, BEGIN CORRECT 
RATE OF DESCENT 

COMPLETE LANDING SETTINGS 

5 MILES FROM TOUCHDOWN 

3 MILES FROM TOUCHDOWN 

r— 

k 

2 1/2 MILES FROM TOUCHDOWN 

2 MILES FROM TOUCHDOWN 

1 1/2 MILES FROM TOUCHDOWN 

1 MILE FROM TOUCHDOWN 

3/4 MILE FROM TOUCHDOWN 

1/2 MILE FROM TOUCHDOWN 

OVER LANDING THRESHOLD 

OVER TOUCHDOWN 

CLEARED FOR TAKE OFF 

COMPLETE TAKE OFF SETTINGS 

COMPLETE DESCENT; CLEARED TO LAND 

ABOVE GLIDE PATH 

BELOW GLIDE PATH 
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ON GLIDE PATH 
SLIGHTLY ABOVE GLIDE PATH 
SLIGHTLY BELOW GLIDE PATH 
WELL ABOVE GLIDE PATH 
WELL BELOW GLIDE PATH 

WIND , KNOTS 

EXECUTE MISSED APPROACH 



The second example is provided by the Synthetic Flight Training 
System, Device 2B24. Automatic audio message alerts are given to a 
student whenever an engine or flight parameter exceeds preset error tol- 
erances. The alert consists of a statement of the parameter which is out- 
of-tolerance. When more than two parameters are out-of-tolerance simul- 
taneously, the alerts are limited to the two highest priority parameters. 
Parameter deviations which continue, are realerted at 10 second intervals. 
In addition, five short coaching messages (10 seconds duration) are provided, 
designed to relieve the instructor of the requirement for continuous verbal 
instructions. The instructor is enabled to override messages as appropri- 
ate to the instructional strategy. 
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3.4. 12 COMMUNICATIONS 

The communications capability at the instructor console involves 
intra-trainer communications and the simulation of radio communications. 

3.4. 12. 1 Intercommunications. The following information requirements 
must be considered in design {as appropriate). 

• Extent of representation — replica of the operational system 
counterparts; the use of actual system communications 
components (e.g. , controls and control panels, headsets, 
microphones, etc*) configured and operated as in the ' 
operational system. 

• Instructor- student communications --selective communica- 
tion with individual students, any combination of students, 
all students, regardless of student control settings. 

• Status displays depicting which students are transmitting, 
receiving and what transmitters are being used; call light 
indication of which student is requesting assistance. 

• Instructor monitoring of all communications channels* 

• Student- to- student communications. 

• Instructor to remote instructor(s)/operator communica- 
tions unavailable to students. 

• Remote instructor to student links. 

. Common audio network- -to place any combination of 
trainee stations in a common network, so that with 
proper tuning/ switching, each student may hear all 
student transmissions or instructor transmissions. 

• Static, background noises, background and simul- 
taneous communications from other sources. 

• Extra-vehicle communications (e.g., messages 
from other units, ground control, etc.). 

3.4. 12.2 Radio Communications. In flight simulators a capability is re- 
quired for instructor monitoring and controlling of radio communications and 
for performing certain ground-to-air control functions (e.g. , GCA, air 
traffic control, etc.). For example, upon hearing a student's call to a 
station, the instructor is able to determine the equipment being used and 
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the station being called. Then, via proper switch activation, he serves as 
ground communicator. As a case in point, when a student calls and the UHF, 
in-range and in-tune light, is on, the instructor can determine which station 
is tuned in (e.g. , by operating the ARC-51 UHF switch and reading the fre- 
quency of the station). This enables him, via the use of a facilities chart, 
to determine that station for which he will act as ground communicator. 

3.4. 12. 3 Communications Controls. The following control capabilities 
should be accounted for: 

• speaker /headset selector and volume controls, 

• alternate action controls to' select the receiver over 
which a student will hear the instructor's voice (e.g. , 
FM, VHF, NAV, UHF), 

• controls and indications for selecting and switching 
all student combinations, 



• controls to activate prerecorded verbal messages (e.g. , 
ground control messages, air traffic control clearances, 
etc.), 

• controls for recording the time of occurrence of errors 
in student communications (i.e., errors in terminology 
or procedures temporally inadequate), 

• controls for recording audio transmissions of the student; 
these controls include: record and playback, indexing and 
fast speed forward/ rewind (see Chapter 3.4.13), 

• controls for selected surface radio/ navigation facilities, 

• controls for channel and frequency selection and in-range 
conditions data between the selected operating equipment 
and the surface facility, 

• layout of communications controls --this involves the 
layout and switching of controls (multi-trainee stations), 
inter-student links, status display of students on the air, 
volume control, "hot mike 11 capability, etc. 
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3.4.13 POST-EXERCISE INSTRUCTIONAL CAPABILITY 

The capability for post-exercise instructional control is a crucial 
design consideration. It is tied-in directly with the' evaluation and the con- 
trol of student performance. Based on the measurement system design for 
the trainer, two instructional functions are served: assessing the progress 
and outcomes of instruction (performance evaluation); and providing infor- 
mation to the student on how the results of his performance conform to 
expectations or norms (knowledge of results). 

The post-exercise instructional capability centers on the critique, 
or debriefing, function. The design issues concern the printout of student 
performance indications, hardware for providing mission critique in the 
trainee station, and special equipments employed in formal critique sessions 
which are used to reconstruct an exercise just completed. Two major 
classes of information requirements must bo considered in providing the 
requisite performance evaluation and knowledge of results capability. These 
relate specifically to the design requirements for 1) hard copy records of 
performance, and 2) graphic assists for formal debriefing sessions. 

3.4. 13. 1 Hard Copy Records. Retrievable permanent records of all 
significant error scoring and time and event happenings should be available 
on demand throughout an exercise. The permanent record serves two 
purposes: the immediate debriefing requirements (for instructional control) 
and the longer term school record keeping (for quality control). Since the 
computer is capable of recording all time and event happenings, and all 
deviations from preset tolerance levels that occur in an exercise, the signi- 
ficant factors that relate to student performance can be provided via hard 
copy printouts. For example, time histories of performance are easily 
obtained, as is status or event information. Frequency data are easily 
assembled and manual control scores (e.g. , integrated error rates) can 
be computed swiftly. Larger segments of performance and terminal (system) 
outputs are also achievable directly. The computer is similarly able to 
combine, integrate and weight scores and compare the results against de- 
fined error tolerance criteria. 

Software programing and hardware design must consider the 
following information requirements . 

a. types of records— Various types of hard copy records are 
available for exercise critique and for school record-keeping purposes. 

. time and event printout- -This provides a record of all 
significant actions taken and the times of accomplish- 
ment throughout the exercise. 
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• error printout- -This provides a record of performance 
errors throughout the exercise. Error indications may 
be in terms of types or groupings , or of all parameters 
which exceed programed tolerances and may include a 
computation of the amount by which each parameter is 
exceeded as a function of time from exercise start. 

• hard copy graphs--This provides a graphic representa- 
tion of performance evaluation for use in off-line analysis 
for quality control. In Device 2B24, for example, the 
program output data for checkride evaluation, provides 
frequency and cumulative time of error by groups of 
students (e.g. , by instructor or by class) for each 
maneuver» for all pertinent parameters without respect 
to the maneuver and for each parameter separately by 
maneuver. Each graph contains separate plots of the 

• same data for at least one comparison group or a refer- 
ence standard group. Also provided is an indication of 
the statistical significance between groups compared. 

• error summaries--This provides short summaries of 
error deviations made during an exercise. The summar- 
ies may depict errors per class, errors achieved per 
mission segment, total error score per exercise, etc. 

A single page summary of performance is useful in the 
post-exercise critique. A performance summary per 
exercise may also be filed for each student; this pro- 
vides an assist in diagnosing performance error and in 
determining subsequent training emphasis. 

b. information to record- -An important feature for design is 
what and how much information to record and in what form. The possi- 
bilities are many, ranging from alphanumeric to graphic information. The 
major concern is for alphanumeric data. Various options are indicated 
below: 

• printout of ail significant events accomplished and 
their time of initiation and completion. 

• printout of performance on selected parameters-- 
For aircraft simulation, it is desirable to record 
performance in the key aircraft and engine param- 
eters* for example: 

- aircraft attitude/ position geometry (heading; 
altitude; airspeed; pitch angle; roil angle; flight 
path and approach slope). 
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- force/rate (turn rate; stick rates; rudder, 

roll, pitch and yaw rates; G leading; and vertical 
velocity) . 

- position of aircraft controls--(stick; rudder and 
throttle positions) 

- discrete events (flaps; landing gear; speed brakes; 
elevator tabs).. 

- instrument readings. 

A desirable capability is the simultaneous recording 
of a given number of the pertinent parameters from 
the total list, on a continuous basis (selectable on 
demand) . 

• printout of deviations from all preprogramed perform- 
ance parameters as the tolerance envelope is exceeded- - 
an example is provided by Device 2F100 (A-7E Aircraft) 
which records the deviation of all selected parameters 
which exceed the preprogramed error tolerances. A 
teletypewriter prints the following information: the 
parameter exceeded, the aircraft position, elapsed 
time, and the direction of deviation. The record is 
available when the parameter returns to within the 
allowable tolerance. 

. printout of all errors committed— for certain training 
situations it is desirable to record all classes of error 
committed, e.g. , by type and frequency or by deviations 
or omissions from * procedural sequence. 

♦u * formattin 8 of retrievable records— A major design issue is 

the formatting requirements in the printouts. The information, which is 
tailored to the type of hard copy record desired, involves: the specific in- 
formation classes to provide, the form of the information presented (e.g. , 
alphanumeric or graphic), and the layout and organization of the selected 
record. Some examples of hard copy formats are shown below to indicate 
the range and extent of the information requirements for design. Figure 31 
presents a summary mission critique printout proposed by Goodyear Aero- 
space Corporation for Device 2-F100 (1971). The printout presents a sum- 

™ r Z o{ ^or exce that is, when the instructor-established tolerances 

for the selected parameters are exceeded. In addition to the identifying 
student and mission information, the following is provided. For each pro- 
cedure, the first line indicates what the student did; the second line indicates 
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what should have been done. All parameters exceeded and frequency of 
occurrence are indicated. From the example in figure 31, an engine fire 
occurred at 20:00 after Comex. The emergency generator should have been 
extended at 21:01, but the student took no action. At 23:00., an ejection was 
begun. The student pulled the face curtain to the second stop when he should " 
have used the first. Throughout the mission, airspeed tolerances were ex- 
ceeded 7 times, altitude 3 times, and position once. -This format gives 
both the instructor and the student a written summary of the errors com- 
mitted during the exercise. Figure 32 provides a sample of an actual 
printout of student performance in Device 2B24. It is part of a summary 
of errors made by a pilot in an automatic checkride mission. The printout 
occurs whenever the student exceeds the tolerable error envelop for a given 
parameter. Each information block pertains to a parameter in which per- 
formance has deviated from the acceptable error envelop. The record is 
self-explanatory providing: time of error occurrence in the mission; desig- 
nated exercise (e.g. , checkride); segment of the mission in which the error 
occurred; the parameter involved (e. g. , altitude); the maximum deviation 
of the error (e.g. , -518 refers to feet below the desired altitude); time 
(seconds) to achieve the maximum error and time (seconds) at maximum 
error; and total time (seconds) out-of- tolerance. 



MISSION CRITIQUE 

NAME D.S. AVIATOR, UNS 17895643 DATE 15 JUNE 1972 

TIME EVENT 

00:20 ENGINE FIRE 

00:21:01 NO ACTION 

EMERGENCY GENERATOR EXTENDED 

00:23 EJECTION SEQUENCE 

00:23:02 FACE CURTAIN SECOND STOP 

FACE CURTAIN FIRST STOP 



PARAMETERS 
AIRSPEED 
ALTITUDE 
POSITION 



NUMBER OF ERRORS 
7 
3 
1 



Figure 3 1 . Summary Printout for Critique . 
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Tint 


I J J 


EXER 


CK.P d 


stun 


AAA 

044 


PARA 


ALT D' 


may nr»; 

nA* Dfcv 




TA MAV 

5tL TO ™AX 


A AA 

000 


CCP AT MAV 

DfcL A I "Aa 


Ann 


SEC OUT TOL 


131 


t r Mr 

Tint 


141 


tXCK 


r l*c n 






D ADA 

r AKA 


to Ar 


MAV ftr*w 
HA* ULV 


A 1 A 

014 


err irt mav 
9Llo I U "AX 


U 1 - 


err at Mav 
S>£L AT "AX 




SEC OUT TOL 


020 


r t mp 


l4o 


tXLK 


CKKU 


crftM 


U4cf 


d «r- a 
rAKft 


ALt L> 


nA«- Utv 


* J « o 


err TA MA*/ 

5tL TO "A/- 


Alt 


err at mav 
5tL AT "AX 


Ann 
000 


CPf* AilT T Al 
5>H* OU I i OL 




TIME 


154 


ExER 


CKPD 


SEGM 


051 


FA 1 . A 


U-AC 


KA"< DEV 


-CO 2 


SEC TO "AX 


■015 


SEC AT *AX 


000 


SEC OUT TOL 


03* 


Tl*E 


155 


EXER 


CKSD 


SEGM 


051 


PARA 


T? AC 


MAX DEV 


&10 


SEC TO MAX 


073 


SEC AT HAX 


000 


SEC OUT TOL 


oso 



Figure 32. Sample printout of student performance 
(error deviation per parameter), Device 
2B24. 



Figure 33 presents a sample printout of pilot performance on the NTDC re- 
search simulator (TRADEC). The recorded segment summarizes perform- 
ance on an automated ground controlled approach (GCA). Figure 34 provide 
a hard copy format that appears useful for performance critique in the Air 
Navigation Trainer, Device 1D23 (Bark, et al 1969). The time and event 
printout is produced for each trainee station in the device. 
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Trainee Station Page 

Time: Event: Heading Change 

Lat. Long. 

WD. . WS 

HD GT 

GS TAS 

AID: 

TACAN TACAN VOR 

118.4 120.3 117.1 



Diagnostics: 

Nav Aids Out of Range: 

120.3 
Out-of-Tol. 



Time: Event: 10-min. Fix 

s 

Lat. . Long. 

WD WS 

HD GT 

GS TAS 

AID: 

TACAN TACAN VOR 

118.4 120.3 113.1 

Time: Event: Annunciator Input 

Lat . Long . 

WD _2ZHHZ WS _zzzzzzi ~ 

HD GT 

GS TAS 

AID: 

VOR TACAN VOR 

113.1 120.3 117.1 



Diagnostics: 
ETA Correct 

Figure 34. Time and event format anticipated for the Air Navigation Trainer, 
(from Bark, et al 1969) 
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In addition to alphanumeric printouts, graphic records pro- 
vide data for performance critique and for off-line analysis (school record 
keeping). A representative example is the graphic recording utilized in 
Device 2F90 (TA-4J aircraft). An eight-channel recorder provides a con- 
tinuous record of the extent of error deviations for selected parameters 
relevant to the specific flight segment being performed. 

d. hard copy selection— A capability is required for initiating 
and terminating hard copy records of performance, on demand. The instruc 
tor must be provided the means for 1) selecting any of the programed per- 
formance variables (parameters, events, system states); 2) selecting the 
sampling rate; and 3) selecting the medium for recording (e.g. , CRT, line 
printer, magnetic tape, typewriter, digital plotter). Controls must also be 
provided for selecting any of a number of programed hard copy formats at 
any time during an exercise and upon completion of a programed exercise. 

e. printout speed— The issue of speed of the hard copy printout 
must be carefully considered. Time and event, error deviation and error 
summary records required for exercise debriefing must be available within 
moments after exercise completion. Similarly, specific performance data 
reqmred during the exercise must be available, as accrued. No stringent 
time constraints are placed on the output of more detailed records for school 
record keeping and for evaluation exercises (e.g. , standardized checkride 
missions). 

f. data storage- -Lengthy training exercises in multi-student 
contexts (e.g. , several hours duration) require a consideration of hard copy 
formats in terms of data storage. The size and number of blocks of data 
must be considered for defined time periods. For example, in Device 1D23, 
Air Navigation Trainer (NTDC 1971) it is reasonable to assume that an 
average of one data block per five minutes of mission type will be stored 
and printed for each student. Thus, thirty-six blocks of data would be re- 
quired for each of 40 students for a three-hour exercise. The printout re- 
quirements for this sizeable amount of data must be correlated with output 
rates of available printing means . 

3.4.13.2 Video Recording and Playback. A record and playback system 
(closed circuit TV used in conjunction with video recording and playback 
equipment) provides, the capability for the student to view his own perform- 
ance (instant playback, if desired) and compare it against some objective 
reference. The instructional purposes served include demonstrations of 
desired performance and the critique of performance completed (knowledge 
of results). 6 
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The information requirements for design include the following: 

• the content (tasks, maneuvers) of performance most suited 
to replay within the finite capacity of the recorder. 

• means for control of the starting point for, and the exiting 
from the playback mode. 

• definition of playback mode time (real, slow times). 

• means for freeze, restart, terminate or bypass portions 
of the playback. 

• definition of standards or expected outcomes for compari- 
son with student performance (knowledge of results derived 
from comparing performance with idealized maneuvers or 
segments, or standard procedures) . 

3.4.13.3 Pictorial Representation. Certain classes of training devices, 
particularly complex team trainers do not possess many automated instruc- 
tional features in design.. Characteristically, automated preprogramed 
mission scenarios are not employed and only a rudimentary measurement 
system is installed. Most often, hard copy printouts of team performance 
are not available. These team training contexts (tactical decision-making, 
operational procedures training) rely heavily on the manual structuring and 
control of the instructional environment, and the post-mission critique is 
based on the observations and judgments of subject matter experts. 

In these situations, special equipments are required to support 
the critique function. Usually, this is in the form of a pictorial reconstruc- 
tion of a just completed exercise. This mission replay, presented to the 
assembled team in a central briefing/ critique area, involves some form of 
visual display projection with or without a correlated audio recording of 
communications. An example is the mission reconstruction utilized in 
Device 14A2, ASROC/ASW Early Attack Weapon System Trainer (NTDC 
1964). A visual display projection of the total mission geometry and critical 
events is provided in support of the mission debriefing. 

The information requirements for a situational display of this 
type for mission reconstruction include the following: 

. plot characteristics (scale, mission area, relative geometry 
of the engagement) 

. vel.'cle symbology (own ship, submarine(s) assist ship(s), 
aircraft) 
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. event symbology (e.g., markers for torpedo firine 
weaponry) e ' 

. track depiction and coding (line traces for specific 
tracks, color coded; weapon traces) 

. speed control of the reconstruction (real time, fast 
time, selective advance or indexing capability) 

. capability for instructor insertion of events into the 
display (e.g. , marker symbols) 

. performance information of error indications (e.g., 
hit or miss, water entry point) 

"E! 16 mOSt current Pictorial displays for debriefing are computer 
generated. For example, Device 2B24, Synthetic Flight Training SvsTem 
employs graphic plots to display ground track history'for each aTr^aft ' 
The recalled ground tracks are presented for full sclle or for any of six 

ee^rann TT* SeCt ° rS - ^ m Sh ° Wn in the fr true 

geographic locations on map backgrounds {i.e., the pertinent geographic 
area » represented at the center of the display with the ground tricks 
J-^iPP^Wr). A zoom mode is also provided to examine por- 

Z T k in detaiL In tMS m ° de * a * iven aircra « i- Posi- 

tioned m the center of the display area, with the track and map background 
of the selected scale appropriately arranged about it. CKground 

f* 4 ;"^ Audio Recording. Multichannel voice recording and playback 
stude„t(s) and instructor channels) provides a continuous recordVvoice 
transmission throughout an exercise. In the recording mode, the. voice 
22? re< : 0rdable somatically whenever a transmitter is acti- 

The Inform eXerC - WitH r6play COntrolled '«« the instructor station. 
The information requirements for design include the following: 

. number of discrete record and playback channels on the 
same tape. 

. number of recorders and the organization of recording 

channels required (for training devices comprising multiple 
independent trainee stations). ' 
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indexing capability of voice recording by time or events, 
in order to permit communications playback in real time, 
or condensed time (omission of blank portions of the tape), 
or selectively for any portion of the tape track involving 
a recorded performance event. 

minimum continuous recording time for each tape. 

adequate forward and rewind search capability in replay 
,mode . 
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3.4.14 HUMAN ENGINEERING DESIGN 



It is tacitly assumed that the basic human engineering design 
standards and military specifications are applicable throughout the device 
design process. Further, it is reasonable to expect that human engineering 
should not be considered as a separate design feature but as an associate 
design requirement integral to the development of the trainer as an instruc- 
tional tool* 



A discussion of human engineering requirements is not provided 
here since the design area is substantial in its own right possessing a well 
developed body of literature with established principles and practices. A 
number of military standards and specifications are current which specify 
the human engineering criteria, data and requirements for the design of the 
training device and its component parts. Principles and guidelines appro- 
priate to trainer design are articulated in the following documents and these 
should be utilized, as appropriate, hi technical approach determination. 

MIL STD 1472A Human Engineering Design Criteria for Military 

Systems, Equipment and Facilities 

MIL T - 23991C Training Devices, Military; General Specification For 

MIL T 82335 Trainers, Flight; General Specification For 

MIL T 9212B Trainer, Flight Simulator, Aircraft, General 

Requirements For 

MIL H 46855 Human Engineering Requirements for Military Systems, 

Equipment and Facilities 

MIL I 82356 Instruments Simulated, For Aircraft Training Devices; 

General Specification For 

The specifics of human engineering design are grouped under the 
following classes of information. Details relevant to each may be found in 
the above referenced standard documents. 



a. Human engineering criteria and data apply to the design of the 
workplace and the man-equipment interfaces as dictated by the instructional 
functions to be performed. Layout and configuration are governed by the 
anthropometry, workspace, maintenance and safety requirements specified 
in the appropriate standard. 

b. Layout should be accomplished in modularized components con- 
sistent with primary and secondary use requirements and good design prac- 
tices. Console configurations should reflect the groupings of displays and 
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controls to accommodate related functions and positioned in terms of fre- 
quency of use and criticality. Console design must also consider flexibility 
m usage (e.g. , accommodating a single instructor or simultaneous use by 
more than one instructor). 

c. Constraints in the workspace and in the equipment layout require 
special consideration, for example, 

• space limitations in room size. 

. integration of components (e.g. , layout which does not 
compromise visibility in a central briefing room; place- 
ment of components to afford direct visual access to all 
trainee stations). 

d. Lighting -Illumination criteria are provided in MIL-STD-1472A 
All panels, recorders, instruments, controls and work surfaces are illu- 
minated as specified. Where different illumination criteria are applicable 
to an area, the highest recommended illumination level pertains. 

e. Acoustical noise-Facility and equipment noise generation and 

E?tt c™°?^? ld bC Controlled in accordance with the requirements of 
MIIj-dTD 147ZA. 

f. Panel markings and displays and controls — The following fea- 
tures must be considered in design: 

• control selection 

• indicator light requirements 

• coding (symbology, color) 

• panel labeling, lettering, and bracketing 

• emergency control markings. 



o 
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SECTION IV 



EVALUATION OF DESIGN 

4. INTRODUCTION 

This section considers two additional human factors topics that are of 
concern in the technical approach documentation. The first topic is the 
test and evaluation requirements involved in the training device acceptance 
process. The second topic is the supporting research requirement. This 
concerns the identification and assessment of the research needed to provide 
technical solution for unresolved or high risk areas, described M in sufficient 
detail to relate technology or methodology improvement to the proposed 
training device 11 (NAVTRADEVCEN INST 3910.5). The issues in each of 
these areas are presented in detail in the remaining two chapters of this 
report. 

4.1 TEST AND EVALUATION 

This chapter provides guidelines for the human factors tests and eval- 
uations to be performed throughout device fabrication to verify the suitability 
of the device as an instructional system. The theme concerns the develop- 
ment of the information requirements that must be satisfied to assure that 
the device performs as intended. The human factors effort subsumes a 
number of issues centering around the following: 

• definition of the expectations and outputs of human factors 
instructional suitability testing. 

• guidance on test plan design, time of accomplishment, and 
the documentation required. 

• identification of the range of testing required and the 
methodology variations ♦ 

• indication of the specific tests and test series (by category) 
to be accomplished. 

The human factors test and evaluation requirements described in this 
chapter are organized to provide inputs useful in the preparation of the per- 
formance specifications for a training device. The emphasis is placed on 
defining the information requirements for determining the capability of the 
device to support the desired training, and on procedures for test guide 
preparation and tnjt execution (the what, how and when of testing). For 
example, information is presented which will enable the writing of precise 
specifications on the kinds of tests and evaluations and demonstrations to 
be accomplished by the contractor at defined points during device fabrication. 
(Currently, a Data Item Description, DD Form 1664, which supports line 
items on DD Form 1423, is used to Specify a desired end product complete 
with instructions for preparation.) 
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if 



4. 1. 1 Perspective . T esting of the instructional capability of a device 
progresses in an iterative sequence beginning with the early conceptual 
evaluation of design in the mockup stage at the factory and culminating in 
the exercising of the device as an integrated training system, on-site at 
the using facility. Further, human factors testing is conducted integral 
with the engineering suitability testing throughout the fabrication cycle. 

Human factors testing should be designed to exercise the total 
capabilities of the device as a training tool. To achieve this, two basic 
types of testing and evaluation are accomplished in the device acceptance 
process. These are: 

Type 1--Static and part-system verification testing . This 
refers to continual time-programed factory testing of perti- 
nent components and 'iubsystems to verify the adequacy of the 
physical and the psychological environment for instructional 
purposes. The goal of these tests, evaluations and demonstra- 
tions is to identify, as early as possible in the fabrication 
process, deviations from the published device specifications 
or emerging discrepancies that will interfere with, or attenuate 
the instructional potential. 

Type 2- -Instructional suitability testing! This refers to dynamic 
operational tests and evaluations to determine eventually that a 
device is instructionally suitable for accomplishing the defined 
training objectives. The testing, accomplished both at the 
factory and on-site, is geared to verifying the validity of the 
device for achieving the purposes of training. 

Figure 35 places the human factors test series in perspective during the 
fabrication process. Since our purpose is to provide guidelines for design, 
no precise time frame is suggested for the testing series since the timing 
requirements are specific to any contemplated device. However, the 
schema is chronological, beginning*with the preparations for conceptual 
demonstrations in the mockup phase and culminating in dynamic testing and 
evaluation once the device is delivered on-site. The human factors test 
series is correlated with the engineering suitability and acceptance test 
series; provision is made for periodic human factors review (as appropriate). 

It should be noted that we are concentrating on a conventional 
approach, beginning with an examination of the design rationale and concep- 
tual demonstrations and proceding to the on-site dynamic testing of the 
device as an instructional system. There are instances, however, where 
a device under procurement serves both as a training and as a research 
vehicle. In this case, the testing may require a problem solving approach 
(i.e. , goals to be achieved) rather than a strict testing rationale since the 
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device may not initially meet all the stated training and performance eval- 
uation objectives. An example of such a situation is found in the suitability 
testing for Device 2B24, Synthetic Flight Training System. This device is 
unique to the Army since, it is not a replacement for existing equipment and 
the training possible with it has heretofor been unavailable to the Army. 
Testing which fails to build on the innovative features of this device would 
probably produce evidence of unsuitability to the Army's requirement. Con- 
versely, testing which exploits the unique training capability of the device 
with the goal of determining its cost effectiveness in training could yield 
quite different results. A complete training program is currently not avail- 
able with the device and substantial program development must be under- 
taken before the device is suitable for operational training (in this respect, 
Device 2B24 is a system under development). But suitability testing cannot 
be delayed until all of the desired automated programs are developed. Ob- 
jective evidence, must be obtained on the workability of the equipment, on 
the suitability of instructor station design and the approach to automated 
training, and on the instructional potential of the device and the manage- 
ability of the developmental work remaining. Thus, the suitability testing 
is geared to identifying the instructor station and automated training and 
performance evaluation features where adjustments may be indicated, while 
determining the suitability of the device independently of such adjustments . 
The testing series comprises three phases: Phase l--determines in a gross 
sense, the workability of the automatic training features of the device; 
Phase II--develops an automated program of instruction (for use in the next 
phase) to test the training functions for which the device is intended; and 
Phase III--involves the conduct a transfer of training study to determine 
the training suitability of the device. 

4.1.2 Approach. The testing and evaluation requirements throughout 
device construction are explicated next. Guidelines are provided for achiev- 
ing the human factors contribution to the device acceptance process. Our 
approach is to organize the test, evaluation and demonstration requirements 
throughout device implementation to assure the delivery of an effective in- 
structional tool. Details are provided on what must be accomplished in the 
chronology of device development within the conceptual -to- dynamic testing 
model outlined in Figure 35. Emphasis is placed on the content, method- 
ology, ana required outputs of the testing series. The material is presented 
in the generic context so as to be applicable to any of the more than twenty 
classes of training- devices currently in the NTDC inventory. In addition, 
guidance is provided concerning the range and extent of human factors test- 
ing to be accomplished and the desired deliverable end products of testing. 
The data of this chapter should effectively support the writing of the test and 
evaluation requirements in the performance specification for a device by 
1) providing the contractor precise directives for the human factors testing 
(by category) and documentation to be accomplished, and 2) installing a 
positive means of control of the acceptance testing procedure throughout 
device fabrication. 
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The remainder of this chapter discusses the test and evaluation 
requirements for: 

• mockup review 

• subsystem verification testing 

• dynamic instructional suitability testing. 

Also, a test plan format of general applicability is proposed for organizing 
the human factors requirements in the device acceptance process. 

4. 1.2. 1 Mockup Review. The mockup inspection is the first formal 
evaluation phase in the development program, occurring prior to the fabri- 
cation of device components. By definition, it represents a closely knit 
coordinative effort between the contractor and the review board designated 
by the procuring agency. Essentially, the device contractor provides a 
mockup of the trainee station, the instructor and operator consoles and 
associated equipment, and an overall layout of the proposed device con- 
figuration. A mockup conference is held, wherein the contractor presents 
the design rationale and the results of the evaluation of the mockup. This 
involves a conceptual demonstration of the validity of the device as an in- 
structional tool including a procedural "walkthrough" of trainer operations 
based on a representative script or scenario. The mockup and a mockup 
evaluation plan is provided in accordance with MIL H 46855 and the contract 
schedule. 

Prior to the formal mockup conference, the contractor develops 
design information sufficient to insure a basis for the evaluation of the 
mockup components. Formal documentation is also identified in the speci- 
fication. Characteristically, a trainer configuration report is specified 
which establishes a baseline for the mockup and review and, in final form, 
reflects the results of the mockup review. Updates are made via change 
sheets, as required. This report, specified via Data Item Description 
(DD form 1664) provides the design criteria and human engineering princi- 
ples and practices to be applied in the design of the general arrangement of 
trainee and instructor stations, and the simulation and control equipment 
groups of the trainer. A representative example of such documentation is 
the Trainer Configuration Report for Device 1D23 (General Electric 1971). 

4.1.2.1.1 Mockup Evaluation Plan. The mockup review is a static 
evaluation ("cardboard" representations, floor plan layouts, verbal demon- 
strations, documentation reports) which serves to verify the hardware 
design layout, the design rationale and instructional system operations. 
It provides a demonstration that the man-equipment combination can accom- 
plish the required operations. The oral presentation by the contractor 
includes: a summary of the human factors analyses (trainee task analysis, 
instructor functions analysis, instructor CRT display formats and content, 



-250- 



NAVTRADEVCEN 7l-C-0<m-l 



performance measurement routines, programed exercises, trainer modes 
of operation, and instructional control capabilities); a checklist verification 
of human engineering requirements; and a conceptual "walkthrough" (via a 
scenario) to demonstrate the adequacy "of the design rationale and trainer 
configuration . 

The specific information requirements that must be satisfied in 
the mockup evaluation plan are outlined next. The contractor provides the 
following documentation and oral demonstrations. 

Human engineering checklist— - Written statements are pre- 
pared which describe the relevant human engineering characteristics that 
effect the operation and use of the device. This forms part of the design 
rationale for the device in that all relevant characteristics are identified 
with a demonstration that compliance with these can be achieved. The 
checklist verification is accomplished in accordance with the design criteria 
published in MIL-STD-1472A and MIL-T-23991 . The checklist identifies 
the human engineering essentials and the adequacy of the interrelations. 
The design issues, as appropriate to instructor and to trainee station design 
include the following: 

• functional groupings of controls and displays 

• anthropometry and workplace layout 

• visual displays 

- illumination 

- information requirements 

- location and arrangement 

- coding 

- ease of use 

• CRT displays 

• indicator lights and legends 

• auditory displays 

• controls — layout and operability 

• labeling 

b. Scenario — A key feature of the mockup evaluation is a con- 
ceptual "walkthrough" of the procedures and sequences in the operation of 
the training system. This constitutes one portion of the oral presentation 
in the mockup review. Based on the training analysis data, a simulated 
mission scenario or an informal script of a representative exercise is 
developed for use in illustrating how the instructor console works and what 
interactions occur between the trainee and the instructor stations. The 
human factors emphasis in these "thought experiments" centers on the in- 
structor station design and layout; little emphasis is given to the trainee 
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station particularly when the station is a replica of the operational system 
(e.g. , an OFT cockpit). The trainee station concern, however, increases 
as more general or universal training functions are provided. ■ With gener- 
alized student stations, the concern includes: the trainee functions provided; 
the fidelity of simulation of key system components; the trainer modes of 
operation; the number and types of controls and displays/and their locations; 
and, for multi-station trainers, the station configuration and instructor-to- 
student ratios* 

The information requirements pertinent to the verification of the 
instructional operations of the device (at the instructor station) range across 
the following. 

• instructor loading- -instructor(s) handling 
of all available instructional options 

• adequacy of instructor station layout 

- all instructional functions represented 

- ease of operation (face validity) 

- control and display groupings by functions 
performed 

- human engineering design (checklist 
verification) 

• adequacy of accomplishing procedures and 
operations in all trainer modes (in the abstract, 
i.e., simulating the motions of performance, 
pointing, verbalizing, etc., in a step-by-step 
accomplishment of each instructional operation 
of the trainer). 

The goal of the mockup review is the acceptance of the proposed final con- 
figuration of the trainer. Based on the contractor demonstrations, the 
procuring agenc 7 is able to: 

• examine the layout and configuration of the 
training device and evaluate conceptually the 
"operability" of the instructional modes via a 
scenario walkthrough (modifications to the 
proposed design are documented in a defined 
end product for this phase of work, e.g., 
final Trainer Configuration Report for the 
device). 
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• adjudge the flexibility of software design to handle 
the available instructional options and enable 
off-line and on-line modifications as necessary. 

. adjudge measurement routines, programed 
exercises. 

• adjudge the adequacy and representativeness of 
the curriculum (e.g. , mission scenarios) to be 
provided. 

. determine the design adequacy for instructor 
station management. 

With the completion of the mockup review and the acceptance 
of the proposed configuration of the training device, the static and part- 
system verification testing of the developing device is initiated. This phase 
is followed by the dynamic instructional suitability testing both at the factory 
and at the training site, which culminates in the acceptance of the training 
device for use in the operational training program. Each of these two test- 
ing phases is discussed next. 

4. 1.2.2 Static and Part-System Verification Testing. Subject to the 
approval of the mockup design review, the contractor establishes and 
conducts a human factors test and evaluation program to verify that the 
varieties of components involved in instructional system operation perform 
as intended. A series of tests are identified (by category) to be accomplished 
as soon as the required components have been fabricated and are available. 
This instructional suitability testing of the developing device is integrated 
with the engineering test sequences . 

For the technical approach documentation, the goal is to identify 
the test plan requirements and to define the series of human factors tests to 
be accomplished. This input to engineering serves as an assist in the sub- 
sequent development of testing requirements in the performance specifica- 
tion for a device. (A generic test plan format which outlines the essential 
elements involved in human factors acceptance testing is provided in para- 
graph 4. 1.2. 1.4.) Thus, the human factors specialist is responsible for 
the following: 

. definition of the relevant tests to be accomplished 
during the device fabrication cycle. 

. definition of the performance requirements or the 
acceptable operations against which test outcomes 
may be compared. 
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The information requirements that must be satisfied during 
this continual factory testing phase are outlined next. Each test requirement 
in the series is undertaken as the equipment or component becomes avail- 
able. Completion of the total series of part- system verification testing is 
the prelude to the subsequent factory and on-site instructional suitability 
testing of the device as an intact training system. The following tests (in 
terms of key issues to evaluate) are recommended to verify the instructional 
capabilities at the instructor console. 

4. 1.2.2. 1 Human Engineering Design of the Instructor Station--The 
configuration and the panel layout of the station and the design of the displays 
and controls are evaluated iteratively based on the mockup design criteria 
and on any subsequent design changes. 

4. 1.2.2.2 Visual Displays--The operability and the use patterns of all 
visual displays at the instructor console are evaluated. The design issues 
to be examined include: 

• location/orientation/ viewing distance in terms 
of intended usage and frequency of usage. 

• accuracy and legibility of the relevant formats . 

4. 1.2.2. 3 CRT, Displays — Tests are conducted of the computer-generated 
displays to verify that the essential information' is provided for instructional 
purposes. The testing issues involve the verification of the following infor- 
mation requirements. 

, flexibility of the display to provide all relevant 
information (student performance and error, 
problem events, scale variations, all problem/ 
display modes) 

» adequate information update 

, rapidity of mode changes (during enroute monitor, 
control, and evaluation) 

• accuracy, reliability- and registration (information 
alignment) of the visual display(s) 

• format requirements , pagination and page 
sequencing 

• multi-formats--high density information display 
and integration; commonality of data locations 
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# continuous display and on-demand display of 
information 

# legibility requirements . 

4. 1.2.2.4 Controls Used for Display Selection And Modification --The 
operability of controls associated with computer generated displays is ex- 
amined for the following: 

» compatibility of manual input means with CRT 
displays (e.g. , keyboard, lightpen) 

0 versatility of display selection (all modes, 
specific events, error and status information, 
scoring information, page sequencing) 

# initiation of desired actions (e.g. , demonstration) 

# modification of preprogramed scenarios (on-line). 

4.1.2.2.5 Software Rrograming-- Testing is required to verify the proper 
operation of automated instructional assists. The content of instructional 
programs is also evaluated (as appropriate). The pertinent software pro- 
graming includes: 

# the content and levels of difficulty of preprogramed 
exercises 

» automated measurement routines (selection of 
measures and scores, combinations of scores, 
performance tolerance envelopes, modification 
of error criteria, performance sampling periods, 
forms of scores displayed, handling of excessive 
student performance errors) 

» adaptive sequencing (adaptive variables, measure- 
ment, adaptive logic, display of the performance 
information) 

4.1.2.2.6 Procedural Performance Display- -Testing is required to 
determine the operability of computer-monitored displays of student pro- 
cedural (checklist) performance. (An example of this type of instructional 
assist is the Trainee Monitoring System (TMS) which provides computer 
monitoring of checklist procedures in flight simulators.) The testing issues 
involve verification of the following. 
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• operation in all modes: on-line (problem mode) 
and off-line (daily readiness check, edit, replay, 
and change memory location modes) 

• legibility of the CRT display 

• demonstration of on-line operability (sequencing 

of procedures, identification of errors of omission/ 
commission, sequencing of CRT pagination (check- 
list of items), instructor intervention capability) 

• operability of controls correlated with the display 
(function keyboard and overlays, cursor employ- 
ment) . 

4. 1.2.2.7 Hard Copy Production — Testing is conducted as early as 
possible to demonstrate the production of performance records. At this 
stage, the purpose is to generate hard-copy printouts, vis-a-vis the design 
rationale (i.e. , error deviations, time and event recording, etc.), estab- 
lish the information formats and verify the .capability to call up or delete 
specific information. Examples of expected outcomes are shown in Figures 
31, 32, 33 and 34 of this report. This operability testing is the prelude to 
the dynamic testing in the next phase which is concerned with the correlation 
between the information recorded and scenario events. 

4. 1.2.2.8 Off- Line Editing — Testing is conducted as early as possible 
to demonstrate the capability for off-line editing of programed scenarios. 
As with item 4.1.2.2.7, the purpose is to evaluate the capability to generate 
modifications to an existing program* An example of such an output is shown 
on page 110 of this report. 

4.1.2.2.9 Specific Stimulus Presentation--Testing is conducted to 
demonstrate the capability to provide specific stimuli (messages displayed 
at prescribed times) which call for observable finite responses by the stu- 
dent. The verification testing issues involve: 

• the presentation of a defined list of programed 
messages presented at precise times in an auto- 
mated sequence 

• student response capability to a message 

. comparison of student response with the situation 

(the information requirements for programing include: 
system status information; vehicle position, movement, 
and direction; rates of resources-expended, etc.) 

• immediate presentation of knowledge of results. 
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4.1.2.2. 10 Trainee Station Design--Tests are also required to verify 
the instructional capabilities of the trainee station. The emphasis is on the 
fidelity required for training and on the evaluation of "trainer peculiar 11 
equipments (i.e. , hardware and associated training functions not present 
in the operator stations in the actual system). Several tests for evaluating 
instructional suitability in the trainee -station are recommended below. 

a. Fidelity of simulation- - Testing is conducted to verify 
that the fidelity of the stimulus or the signal simulation is appropriate to 
the purpose of training for a given device. In situations requiring high 
fidelity simulation, evaluations must be made by a pool of subject matter 
experts (employing an appropriate method of judging) that the information 
displayed is like that found in the operational environment or is judged as 
perceptually equivalent to the operational system. An example of the for- 
mer is the electromagnetic signal environment (EW signatures); an example 
of the latter is motion simulation (in the OFT). In low fidelity situations, 
the evaluation (via an appropriate method of judgment) must verify that the 
level of simulation is adequate for achieving the desired training objectives. 

b. Correlation of control inputs and display indications -- 
Testing is required to determine the appropriateness of vehicle response 
to control inputs with instrument indications and visual display responses. 
In flight simulation, for example, vehicle response to control inputs may 
be evaluated by setting up defined flight situations and requiring expert 
pilots to make specific control inputs. Records a % made of the instrument 
indications resulting from the variation of 3ingle control per trial. Con- 
sistency of device performance and verification of acceptable performance 
(with reference to defined tolerances) can thus be determined. In addition, 
the appropriateness of vehicle response to control inputs (handling qualities) 
can be evaluated via judgments made by experts (e.g. , instructor pilots). 

c. Trainer-peculiar equipments --Testing is recommended 
to verify the operability of equipments placed in the trainee station (not 
found in the operational system). These equipments, designed to enhance 
instructional capability, enable the student to initiate certain instructional 
events (demonstration, parameter freeze, etc.)> or they provide useful 
information about his performance not found in the operational setting. The 
equipments evaluated include: closed circuit TV, trainee monitoring systems 
(computer monitored checklist events), annunciator panels (programed 
messages), problem control panels, and equipments providing visual indi- 
cations of the immediate results of performance (knowledge of results). 

4.1.2.2. 11 Summary - - The purpose of the testing series in this stage is 
to verify the operability of the various components and subsystems that in- 
fluence the instructional capability of the trainer. This level of evaluation 
centers on contractor-testing of the operability of components and subsys- 
tems . Successful demonstrations at this stage serve to minimize the 
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occurrence of discrepancies during the next phase, which is concerned with 
the instructional suitability testing of the assembled trainer. 

4.1.2.3 Instructional Suitability Testing. The final phase of testing is 
begun at the contractor's plant when the training device is assembled, and is 
completed with the on-site acceptance of the device at the training facility by 
the procuring agency. This phase is devoted to operational suitability anal- 
ysis and involves dynamic testing of the fully functioning system. In essence, 
this sequence of testing is accomplished to determine that the contractor has 
satisfied the performance specifications for the device. 

The human factors evaluation (which is part of the operational 
acceptance testing procedure). is conducted to verify the instructional suita- 
bility of the device. The key feature is the employment of a simulated 
mission scenario (or exercise script) to demonstrate that the trainer, as 
an instructional tool, performs "as advertised." The test plan sequence is 
structured to exercise the system in an operational manner, and is there- 
fore concerned with checking -out the scenario. The testing requirements 
and the evaluation sequences which follow are oriented toward training de- 
vices which employ a number of automated instructional assists for enhanc- 
ing the training potential. However, certain tests are identified which 
pertain to devices wherein the setup, control and monitoring of the training 
exercise is accomplished manually. 

4.1.2.3. 1 Scenario Requirements — The preprogramed scenario should 
provide a training sequence (median level of difficulty) which exercises the 
key instructional features of the system. An alternative to this for flight 
simulation is to provide a standard checkride mission. In the case of de- 
vices wherein the training problems are manually installed and controlled 
at the instructor console, a test problem script should be utilized to specify 
the installation of the -types of key events at prescribed times in an exercise 
(i.e. , initial condition parameters setup, insertion of enroute problem 
events with appropriate display indications, and the insertion of emergencies 
and malfunctions). 

4.1.2.3.2 Suitability Analysis— As described earlier, this phase of 
testing is designed to exercise the total capabilities of the fully functioning 
training system in order to demonstrate instructional suitability. "In situ" 
demonstrations are accomplished and judgments made about the following: 

. the adequacy of the instructor stations displays 
and controls for the conduct of training. 

. the extent to which a student or subject matter 
expert is able to proceed through each programed 
sequence in the scenario or accomplish the tasks 
or maneuvers installed in the device. 
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♦ the adjustments required in content, approach 
or design to facilitate accomplishment of train- 
ing objectives • 

In addition, the opinions of qualified operational personnel (instructor pilots, 
sonar and radar operators, etc.) provide additional information on the real- 
istic nature of the installed events and on the handling qualities of simulated 
vehicles. 

Each of the major human factors evaluations to be accom- 
plished in the dynamic setting are identified next. 

a. Instructor station operability .-- The requirement is to de- 
termine that the console(s) functions according to the contractor's documen- 
tation* The specific features include the following: 

• all indicators, controls and display settings 
operate as specified 

• operability of all communications links, intra- 
vehicle and external 

• CRT displays and controls perform as demonstra- 
ted in the previous phase of verification testing 

f operability of on-line instructional controls, and 
controls for off-line (utility) operations 

• hard copy record production capability as demon- 
strated in the previous phase of verification testing. 

b * Setup of initial conditions - -The requirement is to demon- 
strate that the setup of initial conditions in an exercise (either programed, 
manual insertion, or via address of preselected data sets) produces the 
desired system states. This refers to the correctness of the initial con- 
ditions set up vis-a-vis the parameters and values selected (e.g. , vehicle 
position, range, track, speed, etc., and environmental conditions) . 

c* Demonstration of the automated training features of the 
device — This testing is concerned with the hardware and the software pro- 
grams that interface with training functions. The issue is to demonstrate 
that a subject matter expert (e.g. , a proficient pilot) can begin at a defined 
programed entry point in a mission scenario and progress through the pro- 
gram to the intended exit point. The requirement is that the program per-, 
forms as specified and causes the appropriate variations to occur as a func- 
tion of the performer's actions. The validity of the programs may be 
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assessed by 1) the opinions of the participating subject matter experts on 
the technical accuracy of the dynamically developing events in the trainee 
station, and 2) the judgments of human factors specialists on the validity 
of the content and presentation and on the instructional appropriateness of 
the sequence portrayed at the instructor console(s). The automatic train- 
ing features to evaluate are recommended next. 

♦ automatic scheduling programs — this concerns 
changes in the mission displays as a function of 
where the student is in the mission cycle. In 
flight simulation, representative examples in- 
clude the automatic switching from cross-country 
to approach modes, and CRT pagination changes 
(e.g. , automatic switching of air traffic control 
messages as a student proceeds through the 
scenario). 

» automatic scoring — determination is made that 
the device is" scoring and not scoring, as appro- 
priate, via a correlation among the scenario 
time and events, the CRT performance and 
error display(s), and the hardcopy printout. 
This may be accomplished simply by requiring 
the student (or a subject matter expert) to deviate 
deliberately at prescribed times in prescribed 
ways, freeze the problem and observe that 
appropriate scoring information is provided. 

• automated briefing, audio alerts, coaching 
messages --a demonstration is provided that 
all programs commence and end as scheduled 
as a function of student actions and the status 
of the mission. 

, procedural (checklist) performance — an evalua- 
tion is made of the operation of computer-moni- 
tored checklist performance in flight simulators. 
This involves all off-line and on-line modes of 
operation, and includes assessing the display validity 
and the control operations for selecting procedures, 
sequencing the CRT pages and modifying the display. 
(See Chapter 3.4.8 on the Trainee Monitoring 
System ( TMS) . ) 

t specific stimulus presentations --determination is 
made that programed messages which call for 
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specific student behaviors are displayed at 

specified times in the mission cycle. A ; 
sampling of messages is selected and the 
following assessments made: time of message 
onset as programed, length of time displayed, 
response evaluation (based on the response 

input from the trainee station) and adequacy \ 
of knowledge of results presentation. (See 
paragraph 3.4.6.3 on the annunciator panel.) 

instructional options- -tests are made con- 
cerning the flexibility in controlling the exer- 
cise requirements consonant with the develop- 
ment of an instructional strategy for each 
student. These evaluations center on the 
adequacy of the following: 

- display of student performance and error 
information (as accrued) in all relevant 
instructional modes. 

I 

- selection of, and duration of the programed 
demonstration packages. ) 

- control and display capability for remedial 
branching of the student (return to an earlier 

segment or maneuver for reinstruction) . ] 

- error alerts involving computer freeze of j 
the problem when the student exceeds a pre- \ 
set error tolerance; capability for inserting * 
modified error tolerances (A error) is also . 
demonstrated. ; 

- autopilot program is examined as a means j 
for controlling task loading on-the student f 
and as a standard against which student i 
performance may be compared. j 

- insertion of additional events /malfunctions j 
in a programed scenario. j 

I 

• capability in preprogramed scenario to 
manually override an event entry when it 

is position to enter. . 
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, adaptive simulation — a demonstration is 
required that the adaptive sequencing per- 
forms as specified. The test points include 
the following:, selection of the adaptive se- 
quencing mode; task difficulty modification 
as a result of student control performance 
and the display of student performance in- 
formation. The analysis involves a systematic 
introduction of changes in control performance 
(e.g. , via a proficient pilot) to the scenario 
requirements as the basis for assessing the 
extent and speed of change in problem diffi- 
culty level. Judgments of difficulty level by 
a subject matter expert are utilized as well 
as objective indicants of change. 

d. Malfunction programs — malfunction testing is accomplished 
across the range of programed or specified events to determine th^ following: 

• each specific malfunction occurs correctly as 
programed or initiated (i.e. , abrupt or gradual 
occurrence). 

♦ the malfunction is reflected in appropriate instru- 
ment indications and in vehicle control. 

# recovery from the malfunction ban be accomplished 
with appropriate procedures. 

• with similar malfunctions* discriminating cues 
are available to enable the student to identify 
the specific malfunction. 

e. Off-line edit capability - -an evaluation is made of type- 
writer routines for off-line editing of preprogramed problem parameters 
and for the generation of new scenarios. 

f. Response characteristics of the simulator- -an evaluation 
is .made of the* fidelity of simulator response characteristics to the opera- 
tional system counterpart. This assessment is based on the performance 
of subject matter experts in the device. Judgments correlating the device 
performance with their experience in the real world model are used in this 
determination. The issues of concern include the following: 

. handling and response characteristics of the 
device across the range of normal and emer- 
gency control situations provided in the scenario 
or script. 
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• visual attachment displays . 

, atmospheric and environmental conditions 
(turbulence, windshear, engine and rotor 
sounds, etc.) 

• weight and balance program for the vehicle. 

In addition, trainer -peculiar equipments in flight simulators are evaluated 
for ease of use and for the-adequacy of the information presented as an assist 
to instruction. These include: 

• the provision of knowledge of performance 
information. 

• closed circuit TV. 

• annunciator display (specific message 
presentations), 

• trainee monitoring system (TMS, which 
provides checklist procedures displays). 

• problem control panel (means for student 
initiation of device events, such as: 
demonstrations, specific training exercises 
(in the automatic mode), problem halt, 
specific parameter freeze, motion system 
on/off, etc.). 

4. 1.2.3.3 Human Factors Test Plan. The test plan outlines the testing 
and evaluation emphasis to verify device design and to demonstrate that the 
device complies with the specified instructional suitability requirements. 
A generic format is provided in Table 17 which details the elements that 
must be accounted for (minimally) in an effective human factors verification 
testing effort. 
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TABLE 17. OUTLINE FOR HUMAN FACTORS TEST PLAN 



a. Test or Test Sequence 

1. Test identification and stage of testing 

2. Purpose 

3„ Test objectives, concepts and requirements 

b. Test Description 

1 • Summary of the test (initiation and end) 
■2. Emphasis ( demonstration , evaluation, experimental test) 

3. List of equipments involved 

4 # Number of tests to be accomplished 
5. Special considerations 

c. Factors on Which Performance Will Be Evaluated 

1 . Human engineering design 

2. Occurrence of the instructional event or function 

3. Verification of a procedural sequence ' 

4. Time initiation, time latency, performance time 

5. Response occurrence 

6. Accuracy/error tolerance of system response 

7. Flexibility in mode sequencing 

8. Accessibility to displays, controls, equipments 

d. Test Criteria and Measures 



1. Criteria of test accomplishment 

2. Objective measures (units of performance) 

! 

- time 

- frequency 

- accuracy 

- discrepancies/ errors 

- operability 

3. Subjective measures 

- judgments, opinions 
■ - ratings 

4. Points where measures are taken, frequency of occurrence 

5. Reliability of the event, procedure, condition, etc, , that is 
demonstrated. 
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TABLE 17. OUTLINE FOR HUMAN FACTORS TEST PLAN (Continued) 

e. Test and Evaluation Instruments 

1* Checklists 

2. Rating forms 

3. Questionnaires 

4. Instrumentation 

- electrical recording 

- photography 

• graphic recording 

5. Facilities required 

f. Scheduling and Sequencin g of Tests 

1. Test scheduling (time) 

2. Testing sequence and deviations 

3. Conditions under which testing will be accomplished 

4. Length of time each test or test sequence is observed 

g. Data Collection 

1. Data collectors (and number used) 

- subject matter experts 

- human factors specialists 

- other 

2. Detailed data collection procedures 

3. Conditions under which measures are taken or observations made 

4. Number of subjects used 

5. Number of trials or observations made (reliability) 

6. Support requirements 

i- Data Analysis 

1-. Events to be observed and judged 

2. Relations to be tested statistically 

3. Levels of assessment 

- demonstration 

- evaluation (expert judgment 

- experimental test 



3 
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TABLE 17. OUTLINE FOR HUMAN FACTORS TEST PLAN (Continued) 



i. Data Usage or Purpose 

1. Demonstration 

2. Performance evaluation 
3* Failure analysis 

4. Verification of human engineering design of trainer 

5. Verification of instructional suitability 

6* Identification of potential training problems 

j. Reporting Procedures 

1 . Report requirements 

- content 

- number 

- schedule 

2. Results and conclusions 

3. Failure/discrepancy reporting 

4. Design recommendations 



4 * Summary. The human factors suitability testing is predicated on 

several key features. 

a. Human factors testing is integrated with, and is conducted con- 
currently with the engineering acceptance testing. 

b. Testing and evaluation is conducted on an iterative basis, by 
phases, and is accomplished as the equipments, system components and 
the completed device become available. 

c. The testing, on a time continuum, is conveniently organized 
into three consecutive phases beginning with conceptual (paper and pencil) 
demonstrations in the mockup phase, progressing to static and part-system 
testing at the contractor^ factory and culminating in dynamic instructional 
suitability testing both at the factory and finally at the training site once the 
device is delivered and installed. The discussion in Chapter 4. 1 above 
describes the rationale for, and the sequences involved in the test and 
evaluation cycle. 
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d. The final on-site testing is a verification of the dynamic system 
tests conducted at the factory. In addition, the determination is made that 
the complete training system is performing to the specifications (i.e. , for 
multicockpit trainers, all cockpits functioning and integrated in the manage- 
ment of the training) . 
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4.2 SUPPORTING RESEARCH 

In developing the technical approaches to the design of a training 
device, there are instances where the engineering technology is not ade- 
quate to achieve the precision required for effective instruction. These 
cases represent unresolved or high risk areas for des-gn in that simulation 
techniques are not fully developed consonant with the instructional needs, 
or fee feasibility of advanced engineering concepts has not be established 
In the technical approach documentation, formal consideration should be 
given to supporting research requirements in these areas. 

The human factors problem is that the information requirements tend 
to be ill-defined or inexact. Thus, add tional research is needed to improve 
the human factors inputs to the engineering design in these high-risk areas. 
The human factors specialist should participate in identifying the design 
issues involved in maximizing instructional efficiency and in improving the 
information requirements against which candidate technical approaches may 
be compared. Research needs related to the information requirements for 
design may be conveniently grouped into three major categories: 

a - Development of the human factors data base. 

This concerns the definition of the simulation elements required 
to design subsystems in high risk areas including the development of quan- 
titative values for parameters already defined. One major concern in this 
category is the development of design data on the perceptual cues needed 
for vehicle control simulation. An obvious example of this is the continued 
need for the examination of motion cues employed as surrogates for physical 
displacement of flight simulator cbekpits, i.e. , cues that have signal value 
and relevance to the real-world counterpart. This is presumably best 
accomplished within the current state-of-the-art oy simulating the g-forces 
involved, utilizing; acceleration onset cues and washout of acceleration; 
acceleration scaling; and auxiliary signaling means such as pneumatic seat 
cushions, restraining devices and shearing force stimulation. 

*• Quantitative techniques for evaluating technical approaches . 

Development of quantitative techniques may be necessary for iden- 
tifying and selecting engineering alternatives appropriate to specific design 
problems. An example of such a need is provided in paragraph 2 3 2 of 
this report which describes a technique for correlating candidate engineering 
solutions with simulation requirements. 

c - New way's of viewing information requirements fpr design. 

Specifying information requirements needed to support non-conven- 
tional approaches for enhancing traiH$g potential is also a research require- 
ment. The emphasis here is to develop information requirements which will 
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reduce the engineering constraints without compromising instructional 
effectiveness. An illustration of such a human factors research require- 
ment is'to develop an abstract depiction of cartographic information for 
radar landmass display, for example, representing a synthetic world in- 
stead of the real world (display of abstract entities such as a river, a 
cluster of artifacts, etc. , instead of a named river, a named shoreline, 
etc., with their exact characteristics). With abstract depiction, real 
world cartographic information is not required, hence cartographic incre- 
ments may be defined according to need. 

4 * 2 • 1 An Example of a Supporting Research Requirement. Radar land- 
mass representation in low altitude flight simulation is one example of an 
area which could benefit from supporting research. The currently em- 
ployed factored transparency method of simulation suffers in certain re- 
spects due to optical problems and to the difficulty in controlling the size 
and shape of the flying spot (scanning). Resolution is limited by the com- 
bination of spot size, photographic grain size and optics; the limiting 
resolution is tied directly to the size of the problem area (Rubinoff 1971). 
Historically, factored transparencies have been adequate for high altitude 
mission requirements, providing a "birds-eye" view of a large area (e.g. , 
1500 x 500 mile area). However, for displays below 5000 feet, the infor- 
mation presented is too gross for training purposes. For low altitude 
flight, improvements are needed in 1) cartographic information (especially 
for hostile areas), 2) coding technique and 3) readout technique. 

Continued research and development is needed to improve the pre- 
cision of simulation. Various engineering development options are currently 
under investigation. One approach is to improve the factored transparency 
system. This requires a closely coordinated effort designed to improve 
all the relevant components of the optical chain. For example, it is not 
enough to simply reduce the photographic grain size even though the im- 
provement, in itself, may be dramatic. System performance is determined 
by scanner spot size, spot brightness, sensitivity of the detector, and othei^ 
factors, as well as photographic grain size. Therefore, all aspects of the 
system must be improved to a degree commensurate with that achievable 
in the least sensitive element. 

A related but quite distinct problem is the choice of incremental 
units used to encode the basic data. Since the importance of terrain eleva- 
tion is exaggerated at low altitudes, elevation data must be. retrievable from 
the system in small increments. However, the required increments are 
often much smaller than those available from the usual cartographic sources. 

Another promising approach is the digital simulation of radar land- 
mass returns. Here, the photographic analog storage of terrain data is re- 
placed by digital storage in a mass memory. Instead of a flying spot sweep 
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over a transparency, the computer brings forth the appropriate data from 
memory in synchronism with the sweep. These data are converted to analog 
waveforms or profiles and then to radar return signal values. In digital 
simulation, resolution is not directly related to the problem area. This 
characteristic allows the exploitation of inherent terrain redundancies not 
feasible with analog systems. Also, the problem area need not be of fixed 
shape; it may be-patched out of several base maps with almost unlimited 
flexibility. Patching of maps or installing dynamic effects is done under 
computer control (Rubinoff 1971). The primary problems of digital radar 
landmass simulation, however, result from the need to handle massive 
amounts of data, thus requiring tremendous computer capacity (i.e. , nano- 
second computing and gigibit memory). 

In either case, applied research is necessary to specify the infor- 
mation requirements for simulating radar landmass returns for low altitude 
flight. Given good cartographic information and adequate readout and coding 
techniques, the human factors problem is to determine what cues are rele- 
vant in the low altitude context. The question is, what cues does the trainee 
extract in orienting to the terrain? At low altitudes radar reflectivity is 
quite different from that at high altitudes, and significant perceptual distor- 
tions occur e.g. , in shadowing, far-shore brightness, etc.). Techniques 
are not available for calculating these cues, thus, the need for behavioral 
study. 

4 * 2 * 1 * 1 A Proposal for Supporting Research. An attempt to determine 
SSiS?* re * uirements for low altitude radar displays is exemplified in an 
NTDC supporting research proposal. The required resea. ch which is out- 
lined below, is intended to provide results for use in the assign and specifi- 
cation of low altitude radar landmass simulation equipment. 

Research Stud y 

Visual extraction of geographic information from low 
altitude radar displays 

Objective- The task is to define the characteristics of radar returns 
which enable a pilot or radar operator to orient himself to the terrain 
at low altitudes. The operational necessity for low level penetration 
of strike aircraft places increased emphasis on the use of radar in- 
formation for navigation at altitudes of 200 to 1,000 feet. At the 
present time, there is no satisfactory solution to-the problem of 
providing realistic simulated r.idar returns at such low levels 
This is due in part tor a lack of basic information concerning the 
value of certain typical returns as cues to the operator's orientation. 
This study will 1) investigate the returns of prominent terrain and 
cultural features as they appear at various combinations of speed 
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and altitude (up to 1,000 feet), and 2) determine what radar cues are 
closely related to successful performance on the task of geographic 
orientation. 

Approach — A small number of experienced radar operators will be 
asked to indicate their geographic orientation from cues derived 
primarily from the taped radar information (i.e. , video tape re- 
cordings of an operational radar display flown over a prescribed 
ground track at various speeds and altitudes). Radar operators will 
be prebriefed on standard aeronautical charts and then asked to in- 
dicate their location and heading at irregular intervals based upon 
the dynamically displayed radar information. The cue- value of certain 
types of radar returns (e.g. , far-shore brightening) will be evaluated 
by relating the subjects performance (in orienting himself in the 
terrain) to the display details being observed during or just prior to 
his decision. The tape will be stopped at the moment of decision and 
the subject will be asked to a) identify tLose features of the display 
which contributed to his decision, and b) estimate the degree of con- 
fidence to be associated with that decision. These data will provide 
a needed behavioral basis for engineering decisions concerning the 
relative priority of various radar returns_j*yhich could be stored and 
retrieved in radar simulation systems. 

The preliminary data derived from the debriefing techniques 
employed in this phase of study will be extended and refined by the more 
objective method of relating the display details actually being observed to 
the appropriateness of the individual decision. This technique requires 
accurate and continuous measurement of the subject's eye position in syn- 
chrony with the displayed video tapes. 



i 
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